8000 feat: migrate from `deadmanswitch` to Sentry monitors (#585) · python/psf-salt@c785be7 · GitHub
[go: up one dir, main page]
Skip to content

Commit c785be7

Browse files
JacobCoffeeJacobCoffee
authored
feat: migrate from deadmanswitch to Sentry monitors (#585)
1 parent fe2896e commit c785be7

File tree

7 files changed

+56
-59
lines changed

7 files changed

+56
-59
lines changed

‎docs/guides/migration-recipe.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ index 68387c9..7a8ace1 100644
155155
sudo service nginx stop
156156
```
157157
```{note}
158-
Don't forget to pause service checks for both the old and new hosts in things like Dead Man's Snitch, Pingdom, etc.
158+
Don't forget to pause service checks for both the old and new hosts in things like Sentry monitors, Pingdom, etc.
159159
```
160160
4. Ensure that any additional volumes are mounted and in the correct location:
161161
- Check what disks are currently mounted and where: `df`

‎docs/overview.rst

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,11 @@ Pingdom
6363
`Pingdom <https://www.pingdom.com>`_ provides monitoring and complains to us
6464
when services are down.
6565

66+
Sentry
67+
`Sentry <https://sentry.io>`_ is used for error reporting and monitoring of
68+
many services. It also provides Salt highstate cron monitoring, which
69+
notifies us when runs fail over a certain threshold.
70+
6671
PagerDuty
6772
`PagerDuty <https://www.pagerduty.com>`_ is used for on-call rotation for PSF
6873
Infrastructure employees on the front-line, and volunteers as backup.

‎pillar/dev/secrets/sentry.sls

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
project_id: 123456789012345
2+
project_key: deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef
3+
ingest_url: deadbeef.ingest

‎pillar/dev/top.sls

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ base:
88
- tls
99
- users.*
1010
- postgres.clusters
11+
# - secrets.sentry # Uncomment and update sentry secrets if you want to work in dev
1112

1213
'backup-server':
1314
- match: nodegroup

‎salt/_extensions/pillar/dms.py

Lines changed: 0 additions & 48 deletions
This file was deleted.

‎salt/base/auto-highstate.sls

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,27 @@
1-
{% set dms_token = salt["pillar.get"]("deadmanssnitch:token") %}
1+
{% set sentry_enabled = salt["pillar.get"]("project_id") and salt["pillar.get"]("project_key") and salt["pillar.get"]("ingest_url") %}
2+
3+
{% if sentry_enabled %}
4+
curl:
5+
pkg.installed
6+
7+
/usr/local/bin/sentry-checkin.sh:
8+
file.managed:
9+
- source: salt://base/scripts/sentry-checkin.sh.jinja
10+
- template: jinja
11+
- mode: '0755'
12+
- user: root
13+
- group: root
14+
{% endif %}
215

3-
{% if dms_token %}
4-
15m-interval-highstate:
5-
cron.present:
6-
- identifier: 15m-interval-highstate
7-
- name: "timeout 5m salt-call state.highstate >> /var/log/salt/cron-highstate.log 2>&1; curl https://nosnch.in/{{ dms_token }} &> /dev/null"
8-
- minute: '*/15'
9-
{% else %}
1016
15m-interval-highstate:
1117
cron.present:
1218
- identifier: 15m-interval-highstate
13-
- name: "timeout 5m salt-call state.highstate >> /var/log/salt/cron-highstate.log 2>&1"
19+
- name: "{% if sentry_enabled %}/usr/local/bin/sentry-checkin.sh {% endif %}timeout 5m salt-call state.highstate >> /var/log/salt/cron-highstate.log 2>&1"
1420
- minute: '*/15'
15-
{% endif %}
21+
{% if sentry_enabled %}
22+
- require:
23+
- file: /usr/local/bin/sentry-checkin.sh
24+
{% endif %}
1625

1726
/etc/logrotate.d/salt:
1827
{% if grains["oscodename"] == "xenial" %}

‎salt/base/scripts/sentry-checkin.sh.jinja

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#!/bin/bash
2+
3+
MINION_ID="{{ grains['id'] }}"
4+
SENTRY_INGEST_URL="{{ pillar.get('ingest_url', '') }}"
5+
SENTRY_PROJECT_ID="{{ pillar.get('project_id', '') }}"
6+
SENTRY_PROJECT_KEY="{{ pillar.get('project_key', '') }}"
7+
8+
MONITOR_SLUG="salt-highstate-${MINION_ID//./}"
9+
10+
if [ -n "$SENTRY_INGEST_URL" ] && [ -n "$SENTRY_PROJECT_ID" ] && [ -n "$SENTRY_PROJECT_KEY" ]; then
11+
curl -X POST "https://${SENTRY_INGEST_URL}/api/${SENTRY_PROJECT_ID}/cron/${MONITOR_SLUG}/${SENTRY_PROJECT_KEY}/" \
12+
--header 'Content-Type: application/json' \
13+
--data-raw '{"monitor_config": {"schedule": {"type": "crontab", "value": "*/15 * * * *"}, "checkin_margin": 5, "max_runtime": 30, "timezone": "UTC"}, "status": "in_progress"}' &> /dev/null
14+
15+
"$@"
16+
COMMAND_EXIT=$?
17+
18+
if [ $COMMAND_EXIT -eq 0 ]; then
19+
curl "https://${SENTRY_INGEST_URL}/api/${SENTRY_PROJECT_ID}/cron/${MONITOR_SLUG}/${SENTRY_PROJECT_KEY}/?status=ok" &> /dev/null
20+
else
21+
curl "https://${SENTRY_INGEST_URL}/api/${SENTRY_PROJECT_ID}/cron/${MONITOR_SLUG}/${SENTRY_PROJECT_KEY}/?status=error" &> /dev/null
22+
fi
23+
24+
exit $COMMAND_EXIT
25+
else
26+
exit 1
27+
fi

0 commit comments

Comments
 (0)
0