10000 Merge pull request #64 from homeylab/update_readme_http · homeylab/bookstack-file-exporter@6354605 · GitHub
[go: up one dir, main page]
Skip to content

Commit 6354605

Browse files
pchang388pchang388
authored
Merge pull request #64 from homeylab/update_readme_http
update configuration options and descriptions for http_config changes
2 parents 63f10a1 + 045b74e commit 6354605

File tree

5 files changed

+71
-50
lines changed

5 files changed

+71
-50
lines changed

‎README.md

Lines changed: 41 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ Table of Contents
1717
- [Modify Markdown Files](#modify-markdown-files)
1818
- [Object Storage](#object-storage)
1919
- [Minio Backups](#minio-backups)
20+
- [Potential Breaking Upgrades](#potential-breaking-upgrades)
2021
- [Future Items](#future-items)
2122

2223
## Background
@@ -69,20 +70,19 @@ Simple example configuration:
6970
# config.yml
7071
host: "https://bookstack.yourdomain.com"
7172
credentials:
72-
token_id: ""
73-
token_secret: ""
73+
token_id: ""
74+
token_secret: ""
7475
formats: # md only example
7576
- markdown
7677
# - html
7778
# - pdf
7879
# - plaintext
7980
output_path: "bkps/"
8081
assets:
81-
export_images: false
82-
export_attachments: false
83-
modify_markdown: false
84-
export_meta: false
85-
verify_ssl: true
82+
export_images: false
83+
export_attachments: false
84+
modify_markdown: false
85+
export_meta: false
8686
```
8787
8888
### Run via Pip
@@ -205,7 +205,9 @@ Env variables for credentials will take precedence over configuration file optio
205205
**For object storage authentication**, find the relevant sections further down in their respective sections.
206206

207207
### Configuration
208-
_Ensure [Authentication](#authentication-and-permissions) has been set up beforehand for required credentials._ For a simple example to run quickly, refer to the one in the [Using This Application](#using-this-application) section. A full example is also shown below with descriptions. Optionally, look at `examples/` folder of the github repo for more examples.
208+
_Ensure [Authentication](#authentication-and-permissions) has been set up beforehand for required credentials._ For a simple example to run quickly, refer to the one in the [Using This Application](#using-this-application) section.
209+
210+
A full example is also shown below. Optionally, look at `examples/` folder of the github repo for more examples with long descriptions.
209211

210212
For object storage configuration, find more information in their respective sections
211213
- [Minio](#minio-backups)
@@ -218,17 +220,21 @@ Below is an example configuration that shows example values for all possible opt
218220
```yaml
219221
host: "https://bookstack.yourdomain.com"
220222
credentials:
221-
token_id: ""
222-
token_secret: ""
223-
additional_headers:
224-
test: "test"
225-
test2: "test2"
226-
User-Agent: "test-agent"
223+
token_id: ""
224+
token_secret: ""
227225
formats:
228226
- markdown
229227
- html
230228
- pdf
231229
- plaintext
230+
http_config:
231+
verify_ssl: false
232+
timeout: 30
233+
backoff_factor: 2.5
234+
retry_codes: [413, 429, 500, 502, 503, 504]
235+
retry_count: 5
236+
additional_headers:
237+
User-Agent: "test-agent"
232238
minio:
233239
host: "minio.yourdomain.com"
234240
access_key: ""
@@ -243,7 +249,6 @@ assets:
243249
export_attachments: true
244250
modify_markdown: false
245251
export_meta: false
246-
verify_ssl: true
247252
keep_last: 5
248253
run_interval: 0
249254
```
@@ -253,20 +258,25 @@ More descriptions can be found for each section below:
253258

254259
| Configuration Item | Type | Required | Description |
255260
| ------------------ | ---- | -------- | ----------- |
256-
| `host` | `str` | `true` | If `http/https` not specified in the url, defaults to `https`. Use `assets.verify_ssl` to disable certificate checking. |
261+
| `host` | `str` | `true` | If `http/https` not specified in the url, defaults to `https`. Use `http_config.verify_ssl` to disable certificate checking. |
257262
| `credentials` | `object` | `false` | Optional section where Bookstack tokenId and tokenSecret can be specified. Env variable for credentials may be supplied instead. See [Authentication](#authentication) for more details. |
258-
| `credentials.token_id` | `str`| `true` if `credentials` | If `credentials` section is given, this should be a valid tokenId |
259-
| `credentials.token_secret` | `str` | `true` if `credentials`| If `credentials` section is given, this should be a valid tokenSecret |
260-
| `additional_headers` | `object` | `false` | Optional section where key/value for pairs can be specified to use in Bookstack http request headers.
263+
| `credentials.token_id` | `str`| `false` if specified through env var instead, otherwise `true` | A valid Bookstack tokenId. |
264+
| `credentials.token_secret` | `str` | `false` if specified through env var instead, otherwise `true` | A valid Bookstack tokenSecret. |
261265
| `formats` | `list<str>` | `true` | Which export formats to use for Bookstack page content. Valid options are: `["markdown", "html", "pdf", "plaintext"]`|
262266
| `output_path` | `str` | `false` | Optional (default: `cwd`) which directory (relative or full path) to place exports. User who runs the command should have access to read/write to this directory. This directory and any parent directories will be attempted to be created if they do not exist. If not provided, will use current run directory by default. If using docker, this option can be omitted. |
263267
| `assets` | `object` | `false` | Optional section to export additional assets from pages. |
264268
| `assets.export_images` | `bool` | `false` | Optional (default: `false`), export all images for a page to an `image` directory within page directory. See [Backup Behavior](#backup-behavior) for more information on layout |
265269
| `assets.export_attachments` | `bool` | `false` | Optional (default: `false`), export all attachments for a page to an `attachments` directory within page directory. See [Backup Behavior](#backup-behavior) for more information on layout |
266-
| `assets.modify_markdown` | `bool` | `false` | Optional (default: `false`), modify markdown files to replace image links with local exported image paths. This requires `assets.export_images` to be `true` in order to work. See [Modify Markdown Files](#modify-markdown-files) for more information.
267-
| `assets.export_meta` | `bool` | `false` | Optional (default: `false`), export of metadata about the page in a json file |
268-
| `assets.verify_ssl` | `bool` | `false` | Optional (default: `true`), whether or not to check ssl certificates when requesting content from Bookstack host |
269-
| `keep_last` | `int` | `false` | Optional (default: `None`), if exporter can delete older archives. valid values are:<br>- set to `-1` if you want to delete all archives after each run (useful if you only want to upload to object storage)<br>- set to `1+` if you want to retain a certain number of archives<br>- `0` will result in no action done |
270+
| `assets.modify_markdown` | `bool` | `false` | Optional (default: `false`), modify markdown files to replace image links with local exported image paths. This requires `assets.export_images` to be `true` in order to work. See [Modify Markdown Files](#modify-markdown-files) for more information. |
271+
| `assets.export_meta` | `bool` | `false` | Optional (default: `false`), export of metadata about the page in a json file. |
272+
| `http_config` | `object` | `false` | Optional section to override default http configuration. |
273+
| `http_config.verify_ssl` | `bool` | `false` | Optional (default: `false`), whether or not to verify ssl certificates if using https. |
274+
| `http_config.timeout` | `int` | `false` | Optional (default: `30`), set the timeout, in seconds, for http requests. |
275+
| `http_config.retry_count` | `int` | `false` | Optional (default: `5`), the number of http retries after initial failure. |
276+
| `http_config.retry_codes` | `List[int]` | `false` | Optional (default: `[413, 429, 500, 502, 503, 504]`), which http response status codes trigger a retry. |
277+
| `http_config.backoff_factor` | `float` | `false` | Optional (default: `2.5`), set the backoff_factor for http request retries. Default backoff_factor `2.5` means we wait 5, 10, 20, and then 40 seconds (with default `http_config.retry_count: 5`) before our last retry. This should allow for per minute rate limits to be refreshed. |
278+
| `http_config.additional_headers` | `object` | `false` | Optional (default: `{}`), specify key/value pairs that will be added as additional headers to http requests. |
279+
| `keep_last` | `int` | `false` | Optional (default: `0`), if exporter can delete older archives. valid values are:<br>- set to `-1` if you want to delete all archives after each run (useful if you only want to upload to object storage)<br>- set to `1+` if you want to retain a certain number of archives<br>- `0` will result in no action done. |
270280
| `run_interval` | `int` | `false` | Optional (default: `0`). If specified, exporter will run as an application and pause for `{run_interval}` seconds before subsequent runs. Example: `86400` seconds = `24` hours or run once a day. Setting this property to `0` will invoke a single run and exit. Used for basic scheduling of backups. |
271281
| `minio` | `object` | `false` | Optional [Minio](#minio-backups) configuration options. |
272282

@@ -468,7 +478,14 @@ minio:
468478
| `access_key` | `str` | `false` if specified through env var instead, otherwise `true` | Access key for the minio instance |
469479
| `secret_key` | `str` | `false` if specified through env var, otherwise `true` | Secret key for the minio instance |
470480
| `path` | `str` | `false` | Optional, path of the backup to use. Will use root bucket path if not set. `<bucket_name>:/<path>/bookstack-<timestamp>.tgz` |
471-
| `keep_last` | `int` | `false` | Optional (default: `None`), if exporter can delete older archives in minio.<br>- set to `1+` if you want to retain a certain number of archives<br>- `0` will result in no action done |
481+
| `keep_last` | `int` | `false` | Optional (default: `0`), if exporter can delete older archives in minio.<br>- set to `1+` if you want to retain a certain number of archives<br>- `0` will result in no action done |
482+
483+
## Potential Breaking Upgrades
484+
Below are versions that have major changes to the way configuration or exporter runs.
485+
486+
| Start Version | Target Version | Description |
487+
| ------------- | -------------- | ----------- |
488+
| `< 1.4.X` | `1.5.0` | `assets.verify_ssl` has been moved to `http_config.verify_ssl` and the default value has been updated to `false`. `additional_headers` has been moved to `http_config.additional_headers` |
472489

473490
## Future Items
474491
1. ~~Be able to pull images locally and place in their respective page folders for a more complete file level backup.~~

‎bookstack_file_exporter/config_helper/models.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
# pylint: disable=too-few-public-methods
66
class ObjectStorageConfig(BaseModel):
77
"""YAML schema for minio configuration"""
8-
host: str
8+
host: Optional[str] = ""
99
access_key: Optional[str] = ""
1010
secret_key: Optional[str] = ""
1111
bucket: str
@@ -26,7 +26,6 @@ class Assets(BaseModel):
2626
export_attachments: Optional[bool] = False
2727
modify_markdown: Optional[bool] = False
2828
export_meta: Optional[bool] = False
29-
# verify_ssl: Optional[bool] = True
3029

3130
class HttpConfig(BaseModel):
3231
"""YAML schema for user provided http settings"""

‎bookstack_file_exporter/config_helper/remote.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,12 +43,10 @@ def is_valid(self, storage_type: str) -> bool:
4343

4444
def _is_minio_valid(self) -> bool:
4545
"""check if minio config is valid"""
46-
# required values - keys already checked so skip
46+
# required values - keys and bucket already checked so skip
4747
checks = {
48-
"bucket": self.config.bucket,
4948
"host": self.config.host
5049
}
51-
5250
for prop, check in checks.items():
5351
if not check:
5452
log.error("%s is missing from minio configuration and is required", prop)

‎examples/config.yml

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,14 @@
33
# if you put http here, it will try verify=false, not to check certs
44
host: "https://bookstack.mydomain.org"
55
# You could optionally set the bookstack token_id and token_secret here instead of env
6-
# If using env vars instead you can ignore this section
6+
# If using env vars instead you can leave values empty or omit this section
77
credentials:
8-
# set here or as env variable, BOOKSTACK_TOKEN_ID
9-
# env var takes precedence over below
10-
token_id: ""
11-
# set here or as env variable, BOOKSTACK_TOKEN_SECRET
12-
# env var takes precedence over below
13-
token_secret: ""
8+
# set here or as env variable, BOOKSTACK_TOKEN_ID
9+
# env var takes precedence over below
10+
token_id: ""
11+
# set here or as env variable, BOOKSTACK_TOKEN_SECRET
12+
# env var takes precedence over below
13+
token_secret: ""
1414
# supported formats from bookstack below
1515
# specify one or more
1616
formats:
@@ -40,11 +40,18 @@ assets:
4040
# default backoff_factor 2.5 means we wait 5, 10, 20, and then 40 seconds before our last retry
4141
# - this should allow for per minute rate limits to be refreshed
4242
http_config:
43+
# whether or not to verify ssl certificates if using https
4344
verify_ssl: false
45+
# set http timeout in seconds for requests
4446
timeout: 30
47+
# default backoff_factor 2.5 means we wait 5, 10, 20, and then 40 seconds before our last retry
48+
# - this should allow for per minute rate limits to be refreshed
4549
backoff_factor: 2.5
50+
# which status codes trigger retries
4651
retry_codes: [413, 429, 500, 502, 503, 504]
52+
# number of retries
4753
retry_count: 5
54+
# any additional headers to add to http requests
4855
additional_headers:
4956
test: "test"
5057
test2: "test2"

‎examples/minio_config.yml

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,14 @@
33
# if you put http here, it will try verify=false, not to check certs
44
host: "https://bookstack.mydomain.org"
55
# You could optionally set the bookstack token_id and token_secret here instead of env
6-
# If using env vars instead you can omit/comment out this section
6+
# If using env vars instead you can leave values empty or omit this section
77
credentials:
8-
# set here or as env variable, BOOKSTACK_TOKEN_ID
9-
# env var takes precedence over below
10-
token_id: ""
11-
# set here or as env variable, BOOKSTACK_TOKEN_SECRET
12-
# env var takes precedence over below
13-
token_secret: ""
8+
# set here or as env variable, BOOKSTACK_TOKEN_ID
9+
# env var takes precedence over below
10+
token_id: ""
11+
# set here or as env variable, BOOKSTACK_TOKEN_SECRET
12+
# env var takes precedence over below
13+
token_secret: ""
1414
# supported formats from bookstack below
1515
# specify one or more
1616
formats:
@@ -66,23 +66,23 @@ assets:
6666
# optional - can override default http_config
6767
# if not required, you can omit/comment out section
6868
# https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html
69-
# default backoff_factor 2.5 means we wait 5, 10, 20, and then 40 seconds before our last retry
70-
# - this should allow for per minute rate limits to be refreshed
7169
http_config:
70+
# whether or not to verify ssl certificates if using https
7271
verify_ssl: false
72+
# set http timeout in seconds for requests
7373
timeout: 30
74+
# default backoff_factor 2.5 means we wait 5, 10, 20, and then 40 seconds before our last retry
75+
# - this should allow for per minute rate limits to be refreshed
7476
backoff_factor: 2.5
77+
# which status codes trigger retries
7578
retry_codes: [413, 429, 500, 502, 503, 504]
79+
# number of retries
7680
retry_count: 5
81+
# any additional headers to add to http requests
7782
additional_headers:
7883
test: "test"
7984
test2: "test2"
8085
User-Agent: "test-agent"
81-
# After uploading to object storage targets, choose to clean up local files
82-
# delete the archive from local filesystem
83-
# optional
84-
# default = false if omitted
85-
clean_up: true
8686
# optional if specified exporter can delete older archives
8787
# valid values are:
8888
# set to -1 if you want to delete all archives after each run

0 commit comments

Comments
 (0)
0