8000 Implement configurations for WAL-G #145
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Let postgres Linux user execute wal-g as wal-g Linux user
- specifically to groups that the postgres Linux user are a part of
- not enabled however
- Also, use nohup
- to allow looping through
…s into chore/walg_enhancements
darora
approved these changes
Mar 30, 2022
|
|
||
| set -euo pipefail | ||
|
|
||
| WALG_SENTINEL_USER_DATA="{ \"backup_id\": $1, \"project_id\": $2 }" nohup wal-g backup-push /var/lib/postgresql/data --config /etc/wal-g/config.json --verify >> /var/log/wal-g/backup-push.log 2>&1 & |
There was a problem hiding this comment.
just use systemd-cat? Do we expect this to be extremely verbose?
There was a problem hiding this comment.
It can get quite verbose depending on the size of the database. For example, this is a full backup done for a ~ 15 GB database.
INFO: 2022/03/30 16:24:20.486917 Selecting the latest backup as the base for the current delta backup...
INFO: 2022/03/30 16:24:20.616928 Reached max delta steps. Doing full backup.
INFO: 2022/03/30 16:24:20.639508 Calling pg_start_backup()
INFO: 2022/03/30 16:24:20.704709 Starting a new tar bundle
INFO: 2022/03/30 16:24:20.704745 Walking ...
INFO: 2022/03/30 16:24:20.704972 Starting part 1 ...
INFO: 2022/03/30 16:24:39.139675 Finished writing part 1.
INFO: 2022/03/30 16:24:39.139706 Starting part 2 ...
INFO: 2022/03/30 16:24:56.362484 Finished writing part 2.
INFO: 2022/03/30 16:24:56.362518 Starting part 3 ...
INFO: 2022/03/30 16:25:12.891977 Finished writing part 3.
INFO: 2022/03/30 16:25:12.892011 Starting part 4 ...
INFO: 2022/03/30 16:25:29.653868 Finished writing part 4.
INFO: 2022/03/30 16:25:29.653915 Starting part 5 ...
INFO: 2022/03/30 16:25:45.906419 Finished writing part 5.
INFO: 2022/03/30 16:25:45.906558 Starting part 6 ...
INFO: 2022/03/30 16:26:10.879299 Finished writing part 6.
INFO: 2022/03/30 16:26:10.879339 Starting part 7 ...
INFO: 2022/03/30 16:26:27.899468 Finished writing part 7.
INFO: 2022/03/30 16:26:27.899497 Starting part 8 ...
INFO: 2022/03/30 16:26:44.314572 Finished writing part 8.
INFO: 2022/03/30 16:26:44.314606 Starting part 9 ...
INFO: 2022/03/30 16:27:00.893701 Finished writing part 9.
INFO: 2022/03/30 16:27:00.893731 Starting part 10 ...
INFO: 2022/03/30 16:27:17.346363 Finished writing part 10.
INFO: 2022/03/30 16:27:17.346395 Starting part 11 ...
INFO: 2022/03/30 16:27:33.721219 Finished writing part 11.
INFO: 2022/03/30 16:27:33.721251 Starting part 12 ...
INFO: 2022/03/30 16:27:50.275346 Finished writing part 12.
INFO: 2022/03/30 16:27:50.275377 Starting part 13 ...
INFO: 2022/03/30 16:28:06.648361 Finished writing part 13.
INFO: 2022/03/30 16:28:06.648392 Starting part 14 ...
INFO: 2022/03/30 16:28:17.527654 Finished writing part 14.
INFO: 2022/03/30 16:28:17.527697 Starting part 15 ...
INFO: 2022/03/30 16:28:28.318634 Finished writing part 15.
INFO: 2022/03/30 16:28:28.318662 Starting part 16 ...
INFO: 2022/03/30 16:28:35.075051 Packing ...
INFO: 2022/03/30 16:28:35.076358 Finished writing part 16.
INFO: 2022/03/30 16:28:35.400641 Starting part 17 ...
INFO: 2022/03/30 16:28:35.400675 /global/pg_control
INFO: 2022/03/30 16:28:35.402910 Finished writing part 17.
INFO: 2022/03/30 16:28:35.403969 Calling pg_stop_backup()
INFO: 2022/03/30 16:28:36.446093 Starting part 18 ...
INFO: 2022/03/30 16:28:36.453716 backup_label
INFO: 2022/03/30 16:28:36.453859 tablespace_map
INFO: 2022/03/30 16:28:36.453963 Finished writing part 18.
INFO: 2022/03/30 16:28:36.580489 Wrote backup with name base_0000000100000007000000D4
Also, would want to split the logs for the different wal-g sub-commands used:
- backup-push
- backup-fetch
- wal-push
- wal-fetch
|
|
||
| set -euo pipefail | ||
|
|
||
| sed -i "s/.*archive_mode/#archive_mode/" /etc/postgresql/postgresql.conf |
There was a problem hiding this comment.
Instead of modifying this conf file, what do you think of including a separate file, and writing to it/emptying it out to enable/disable wal-g?
There was a problem hiding this comment.
Updated to include a separate file housing parameters pertaining to wal-g. Scripts are pointed to this new file instead to modify these parameters.
| @@ -24,7 +24,7 @@ | |||
| # Download WAL-G | |||
| - name: wal-g - download latest version | |||
| git: | |||
| repo: https://github.com/darora/wal-g.git | |||
hf
pushed a commit
that referenced
this pull request
Jan 23, 2023
* chore: create wal-g Linux user - Let postgres Linux user execute wal-g as wal-g Linux user * Revert WAL-G backport * chore: give read access to PG data directory to groups - specifically to groups that the postgres Linux user are a part of * update configs in postgresql.conf - not enabled however * WAL-G specific scripts to be triggered via the admin API * Create config directory for wal-g * chore: add logging for wal-g processes * ensure bindir when installing pgsodium works for EC2 builds * triggere dockerbuild * Run backup in background * Inject WALG_SENTINEL_USER_DATA during backups - Also, use nohup * Add scripts for enabling and disabling walg * removed unused scripts * bump versioning * cleanup * Move logrotate to its own dir - to allow looping through * cleanup: loop through creation of log files * set up logrotate for all WAL-G log files * Ensure backup is pointing to the correct log file * create postgres linux group * ensure that it's config.json, not .env * allow admin API to edit /etc/wal-g/config.json * add whitespace * script to complete wal-g restoration * update adminapi * remove bindir in setup-system * create wal-g group and move up setting up of wal-g * create wal-g log files towards the end * Ensure /var/lib/postgresql/data is owned by postgres:postgres * fix transfer of file found in subdirectories * set primary groups for postgres and wal-g linux users * newline * modify wal-g related parameters in separate file * bump to 14.1.0.21 * more newlines
damonrand
pushed a commit
to cepro/postgres
that referenced
this pull request
Jun 15, 2025
* chore: create wal-g Linux user - Let postgres Linux user execute wal-g as wal-g Linux user * Revert WAL-G backport * chore: give read access to PG data directory to groups - specifically to groups that the postgres Linux user are a part of * update configs in postgresql.conf - not enabled however * WAL-G specific scripts to be triggered via the admin API * Create config directory for wal-g * chore: add logging for wal-g processes * ensure bindir when installing pgsodium works for EC2 builds * triggere dockerbuild * Run backup in background * Inject WALG_SENTINEL_USER_DATA during backups - Also, use nohup * Add scripts for enabling and disabling walg * removed unused scripts * bump versioning * cleanup * Move logrotate to its own dir - to allow looping through * cleanup: loop through creation of log files * set up logrotate for all WAL-G log files * Ensure backup is pointing to the correct log file * create postgres linux group * ensure that it's config.json, not .env * allow admin API to edit /etc/wal-g/config.json * add whitespace * script to complete wal-g restoration * update adminapi * remove bindir in setup-system * create wal-g group and move up setting up of wal-g * create wal-g log files towards the end * Ensure /var/lib/postgresql/data is owned by postgres:postgres * fix transfer of file found in subdirectories * set primary groups for postgres and wal-g linux users * newline * modify wal-g related parameters in separate file * bump to 14.1.0.21 * more newlines
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Primary changes
--allow-group-accessoption. This allows other Linux users within the same groups as the cluster owner to have read access to the data directory. ref.wal-g. ThepostgresLinux user is given sudo access to the execute the wal-g binary as thewal-gLinux user.