8000 Implement configurations for WAL-G (#145) · supabase/postgres@1c60ef0 · GitHub
[go: up one dir, main page]
Skip to content

Commit 1c60ef0

Browse files
dragarciahf
dragarcia
authored and
hf
committed
Implement configurations for WAL-G (#145)
* chore: create wal-g Linux user - Let postgres Linux user execute wal-g as wal-g Linux user * Revert WAL-G backport * chore: give read access to PG data directory to groups - specifically to groups that the postgres Linux user are a part of * update configs in postgresql.conf - not enabled however * WAL-G specific scripts to be triggered via the admin API * Create config directory for wal-g * chore: add logging for wal-g processes * ensure bindir when installing pgsodium works for EC2 builds * triggere dockerbuild * Run backup in background * Inject WALG_SENTINEL_USER_DATA during backups - Also, use nohup * Add scripts for enabling and disabling walg * removed unused scripts * bump versioning * cleanup * Move logrotate to its own dir - to allow looping through * cleanup: loop through creation of log files * set up logrotate for all WAL-G log files * Ensure backup is pointing to the correct log file * create postgres linux group * ensure that it's config.json, not .env * allow admin API to edit /etc/wal-g/config.json * add whitespace * script to complete wal-g restoration * update adminapi * remove bindir in setup-system * create wal-g group and move up setting up of wal-g * create wal-g log files towards the end * Ensure /var/lib/postgresql/data is owned by postgres:postgres * fix transfer of file found in subdirectories * set primary groups for postgres and wal-g linux users * newline * modify wal-g related parameters in separate file * bump to 14.1.0.21 * more newlines
1 parent c1c0647 commit 1c60ef0

19 files changed

+180
-24
lines changed

‎ansible/files/admin_api_scripts/commence_walg_backup.sh

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#! /usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
WALG_SENTINEL_USER_DATA="{ \"backup_id\": $1, \"project_id\": $2 }" nohup wal-g backup-push /var/lib/postgresql/data --config /etc/wal-g/config.json --verify >> /var/log/wal-g/backup-push.log 2>&1 &
6+
7+
echo "WAL-G backup job commenced"

‎ansible/files/admin_api_scripts/complete_walg_restore.sh

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#! /usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
# disable 169.254.169.254 for postgres
6+
sed -i "/#\\sdon't\\sdelete\\sthe\\s'COMMIT'/ i -I OUTPUT 1 --proto tcp --destination 169.254.169.254 --match owner --uid-owner postgres --jump REJECT\\n" /etc/ufw/before.rules
7+
ufw reload
8+
9+
# move config file to its final location and change its ownership
10+
mv /etc/postgresql/wal-g-config.json /etc/wal-g/config.json
11+
chown wal-g:wal-g /etc/wal-g/config.json
12+
13+
14+
# disable recovery commands in the event of a restart
15+
sed -i "s/.*restore_command/#restore_command/" /etc/postgresql-custom/wal-g.conf
16+
sed -i "s/.*recovery_target_time/#recovery_target_time/" /etc/postgresql-custom/wal-g.conf
17+
sed -i "s/.*recovery_target_action/#recovery_target_action/" /etc/postgresql-custom/wal-g.conf
18+
19+
# enable archive_command
20+
sed -i "s/.*archive_mode/archive_mode/" /etc/postgresql-custom/wal-g.conf
21+
sed -i "s/.*archive_command/archive_command/" /etc/postgresql-custom/wal-g.conf
22+
sed -i "s/.*archive_timeout/archive_timeout/" /etc/postgresql-custom/wal-g.conf
23+
24+
systemctl restart postgresql
25+
26+
echo "Cleanup post WAL-G restoration complete"

‎ansible/files/admin_api_scripts/disable_walg.sh

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#! /usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
sed -i "s/.*archive_mode/#archive_mode/" /etc/postgresql-custom/wal-g.conf
6+
sed -i "s/.*archive_command/#archive_command/" /etc/postgresql-custom/wal-g.conf
7+
sed -i "s/.*archive_timeout/#archive_timeout/" /etc/postgresql-custom/wal-g.conf
8+
9+
systemctl restart postgresql
10+
11+
echo "WAL-G successfully disabled"

‎ansible/files/admin_api_scripts/enable_walg.sh

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#! /usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
sed -i "s/.*archive_mode/archive_mode/" /etc/postgresql-custom/wal-g.conf
6+
sed -i "s/.*archive_command/archive_command/" /etc/postgresql-custom/wal-g.conf
7+
sed -i "s/.*archive_timeout/archive_timeout/" /etc/postgresql-custom/wal-g.conf
8+
9+
systemctl restart postgresql
10+
11+
echo "WAL-G successfully enabled"

‎ansible/files/grow_fs.sh renamed to ‎ansible/files/admin_api_scripts/grow_fs.sh

File renamed without changes.

‎ansible/files/logrotate-postgres-csv renamed to ‎ansible/files/logrotate_config/logrotate-postgres-csv.conf

File renamed without changes.

‎ansible/files/logrotate-postgres renamed to ‎ansible/files/logrotate_config/logrotate-postgres.conf

File renamed without changes.

‎ansible/files/logrotate_config/logrotate-walg.conf

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
/var/log/wal-g/*.log {
2+
size 50M
3+
rotate 3
4+
copytruncate
5+
delaycompress
6+
compress
7+
notifempty
8+
missingok
9+
}

‎ansible/files/postgresql_config/custom_walg.conf.j2

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# - Archiving -
2+
3+
#archive_mode = on
4+
#archive_command = 'sudo -u wal-g wal-g wal-push %p --config /etc/wal-g/config.json >> /var/log/wal-g/wal-push.log 2>&1'
5+
#archive_timeout = 120
6+
7+
8+
# - Archive Recovery -
9+
10+
#restore_command = 'sudo -u wal-g wal-g wal-fetch "%f" "%p" --config /etc/wal-g/config.json >> /var/log/wal-g/wal-fetch.log 2>&1'
11+
12+
13+
# - Recovery Target -
14+
15+
#recovery_target_time = ''
16+
#recovery_target_action = 'promote'

‎ansible/files/postgresql_config/postgresql.conf.j2

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -791,6 +791,9 @@ jit_provider = 'llvmjit' # JIT library to use
791791
# User-supplied custom parameters, override any automatically generated ones
792792
#include = '/etc/postgresql-custom/custom-overrides.conf'
793793

794+
# WAL-G specific configurations
795+
#include = '/etc/postgresql-custom/wal-g.conf'
796+
794797
#------------------------------------------------------------------------------
795798
# CUSTOMIZED OPTIONS
796799
#------------------------------------------------------------------------------

0 commit comments

Comments
 (0)
0