8000 Merge pull request #122 from opensafely-core/madwort/otel-grafana · opensafely-core/sysadmin@2090ae4 · GitHub
[go: up one dir, main page]
Skip to content

Commit 2090ae4

Browse files
madwortmadwort
authored
Merge pull request #122 from opensafely-core/madwort/otel-grafana
feat: duplicate otel data to Grafana
2 parents 4373d76 + 5bbb93e commit 2090ae4

File tree

8 files changed

+126
-24
lines changed

8 files changed

+126
-24
lines changed

‎services/otel-gateway/Dockerfile

+4-1
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,16 @@ LABEL org.opencontainers.image.authors="tech@opensafely.org" \
1212
org.opencontainers.image.source="https://github.com/opensafely-sysadmin"
1313

1414
# default config, can be overridden at runtime
15-
ENV ENDPOINT="https://api.honeycomb.io"
15+
ENV HONEYCOMB_ENDPOINT="https://api.honeycomb.io"
16+
ENV GRAFANA_ENDPOINT="https://otlp-gateway-prod-gb-south-0.grafana.net/otlp"
1617
ENV METRICS_DATASET="jobrunner-metrics"
1718
ENV LOG_LEVEL="info"
1819

1920
# these must be provided at runtime
2021
# ENV HONEYCOMB_KEY
2122
# ENV BASIC_AUTH_USER
2223
# ENV BASIC_AUTH_PASSWORD
24+
# ENV GRAFANA_INSTANCE_ID
25+
# ENV GRAFANA_API_TOKEN
2326

2427
COPY config.yaml /etc/otelcol-contrib/config.yaml

‎services/otel-gateway/README.md

+5-3
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,16 @@ traffic and basic auth.
2424

2525
### Integration Tests
2626

27-
You will need a valid API key for the developement environment
27+
You will need valid API keys for the development environment
2828

2929
```
3030
export HONEYCOMB_KEY=...
31+
export GRAFANA_INSTANCE_ID=...
32+
export GRAFANA_API_TOKEN=...
3133
just test-integration
3234
```
3335

34-
This will print a link where you can go and view the data in the Honeycomb UI.
36+
This will print a link where you can go and view the data in the Honeycomb UI. It should also send data to Grafana.
3537

3638

3739
## Deploy to dokku
@@ -62,7 +64,7 @@ dokku proxy:ports-add otel-gateway https:443:4318
6264
dokku letsencrypt:enable otel-gateway
6365

6466
# secrets
65-
dokku config:set otel-gateway HONEYCOMB_KEY=... BASIC_AUTH_USER=... BASIC_AUTH_PASSWORD=...
67+
dokku config:set otel-gateway HONEYCOMB_KEY=... BASIC_AUTH_USER=... BASIC_AUTH_PASSWORD=... GRAFANA_INSTANCE_ID=... GRAFANA_API_TOKEN=...
6668

6769
```
6870

‎services/otel-gateway/config.yaml

+14-5
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@ extensions:
55
htpasswd:
66
inline: |
77
${BASIC_AUTH_USER}:${BASIC_AUTH_PASSWORD}
8+
basicauth/otlp:
9+
client_auth:
10+
username: ${GRAFANA_INSTANCE_ID}
11+
password: ${GRAFANA_API_TOKEN}
812

913
receivers:
1014
otlp:
@@ -21,30 +25,35 @@ exporters:
2125
logLevel: "${LOG_LEVEL}"
2226

2327
otlphttp/traces:
24-
endpoint: "${ENDPOINT}"
28+
endpoint: "${HONEYCOMB_ENDPOINT}"
2529
headers:
2630
x-honeycomb-team: "${HONEYCOMB_KEY}"
2731
# Do not set a dataset - let forwarding application set that.
2832

2933
# for some reason, metrics requires an explicit dataset, so we have
3034
# a separate exporter config for that
3135
otlphttp/metrics:
32-
endpoint: "${ENDPOINT}"
36+
endpoint: "${HONEYCOMB_ENDPOINT}"
3337
headers:
3438
x-honeycomb-team: "${HONEYCOMB_KEY}"
3539
x-honeycomb-dataset: "${METRICS_DATASET}"
3640

41+
otlphttp/grafana:
42+
auth:
43+
authenticator: basicauth/otlp
44+
endpoint: "${GRAFANA_ENDPOINT}"
45+
3746
service:
3847
telemetry:
3948
logs:
4049
level: "${LOG_LEVEL}"
41-
extensions: [basicauth/server]
50+
extensions: [basicauth/server, basicauth/otlp]
4251
pipelines:
4352
traces:
4453
receivers: [otlp]
4554
processors: [batch]
46-
exporters: [otlphttp/traces, logging]
55+
exporters: [otlphttp/traces, otlphttp/grafana, logging]
4756
metrics:
4857
receivers: [otlp]
4958
processors: [batch]
50-
exporters: [otlphttp/metrics, logging]
59+
exporters: [otlphttp/metrics, otlphttp/grafana, logging]

‎services/otel-gateway/dotenv-sample

+2
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
11
BASIC_AUTH_USER="test"
22
BASIC_AUTH_PASSWORD="test"
33
HONEYCOMB_KEY="testkey"
4+
GRAFANA_INSTANCE_ID="012345"
5+
GRAFANA_API_TOKEN="glc_aoeuhtnsqjkxbmwv1234567890="

‎services/otel-gateway/justfile

+47-10
Original file line numberDiff line numberDiff line change
@@ -19,46 +19,83 @@ run *args: build
1919
-e BASIC_AUTH_USER=$BASIC_AUTH_USER \
2020
-e BASIC_AUTH_PASSWORD=$BASIC_AUTH_PASSWORD \
2121
-e HONEYCOMB_KEY \
22+
-e GRAFANA_INSTANCE_ID \
23+
-e GRAFANA_API_TOKEN \
2224
{{ args }} {{ IMAGE_NAME }}
2325

2426

25-
# run integration test. You will need a HONEYCOMB_KEY set in the environment
27+
# run integration test. You will need keys for Honeycomb & Grafana set in the environment
2628
test-integration: venv
2729
#!/bin/bash
2830
set -euo pipefail
2931

3032
{{ just_executable() }} run -d -e LOG_LEVEL=debug -p 4318:4318
3133
{{ just_executable() }} run-python tests.py
32-
echo "Data sent to honeycomb"
34+
echo "Data sent to honeycomb and grafana"
3335
echo "https://ui.honeycomb.io/bennett-institute-for-applied-data-science/environments/development/datasets/otel-gateway-tests?query=%7B%22time_range%22%3A600%2C%22granularity%22%3A0%2C%22breakdowns%22%3A%5B%5D%2C%22calculations%22%3A%5B%5D%2C%22orders%22%3A%5B%5D%2C%22havings%22%3A%5B%5D%2C%22limit%22%3A100%7D"
3436

3537

36-
# spin up a pretend honeycomb and run tests
37-
test-ci: venv
38+
_mock_honeycomb_start:
3839
#!/bin/bash
3940
set -euo pipefail
4041

41-
mkdir -p exported
42+
mkdir -p exported/honeycomb
4243

4344
# run a different instance of a collector as a test endpoint
4445
docker rm --force mock-honeycomb 2>/dev/null || true
4546
docker run -d -p 4319:4318 --name mock-honeycomb -u "$(id -u):$(id -g)" \
46-
-v $PWD/test-config.yaml:/etc/otelcol-contrib/config.yaml \
47-
-v $PWD/exported:/exported \
47+
-v $PWD/mock-honeycomb-config.yaml:/etc/otelcol-contrib/config.yaml \
48+
-v $PWD/exported/honeycomb:/exported \
4849
otel/opentelemetry-collector-contrib:0.62.1
4950

5051
test "$(docker inspect mock-honeycomb -f '{{{{.State.Status}}')" == "running" || { docker logs mock-honeycomb; exit 1; }
5152

52-
export ENDPOINT="http://host.docker.internal:4319"
53+
_mock_honeycomb_stop:
54+
docker stop mock-honeycomb
55+
56+
_mock_grafana_start:
57+
#!/bin/bash
58+
set -euo pipefail
59+
60+
mkdir -p exported/grafana
61+
62+
# run a different instance of a collector as a test endpoint
63+
docker rm --force mock-grafana 2>/dev/null || true
64+
docker run -d -p 4320:4318 --name mock-grafana -u "$(id -u):$(id -g)" \
65+
-v $PWD/mock-grafana-config.yaml:/etc/otelcol-contrib/config.yaml \
66+
-v $PWD/exported/grafana:/exported \
67+
-e GRAFANA_INSTANCE_ID \
68+
-e GRAFANA_API_TOKEN \
69+
otel/opentelemetry-collector-contrib:0.62.1
70+
71+
test "$(docker inspect mock-grafana -f '{{{{.State.Status}}')" == "running" || { docker logs mock-grafana; exit 1; }
72+
73+
_mock_grafana_stop:
74+
docker stop mock-grafana
75+
76+
# run tests against mock upstream servers
77+
test-ci: venv _mock_honeycomb_start _mock_grafana_start && _mock_honeycomb_stop _mock_grafana_stop
78+
#!/bin/bash
79+
set -euo pipefail
80+
81+
export HONEYCOMB_ENDPOINT="http://host.docker.internal:4319"
82+
export GRAFANA_ENDPOINT="http://host.docker.internal:4320"
5383

5484
# run otel-gateway pointing at the test endpoint
55-
{{ just_executable() }} run -d -e ENDPOINT -e LOG_LEVEL=debug -p 4318:4318 --add-host=host.docker.internal:host-gateway
85+
{{ just_executable() }} run -d \
86+
-e HONEYCOMB_ENDPOINT \
87+
-e GRAFANA_ENDPOINT \
88+
-e GRAFANA_INSTANCE_ID \
89+
-e GRAFANA_API_TOKEN \
90+
-e LOG_LEVEL=debug \
91+
-p 4318:4318 \
92+
--add-host=host.docker.internal:host-gateway
5693

5794
test "$(docker inspect otel-gateway -f '{{{{.State.Status}}')" == "running" || { docker logs otel-gateway; exit 1; }
5895
{{ just_executable() }} run-python -m pytest tests.py
5996

6097
docker stop otel-gateway
61-
docker stop mock-honeycomb
98+
6299

63100
# run a python script in the correct environment
64101
run-python *args: venv

‎services/otel-gateway/mock-grafana-config.yaml

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
2+
extensions:
3+
basicauth/server:
4+
htpasswd:
5+
inline: |
6+
${GRAFANA_INSTANCE_ID}:${GRAFANA_API_TOKEN}
7+
8+
receivers:
9+
otlp:
10+
protocols:
11+
http:
12+
auth:
13+
authenticator: basicauth/server
14+
15+
processors:
16+
batch:
17+
18+
exporters:
19+
logging:
20+
logLevel: debug
21+
file/traces:
22+
path: /exported/traces.json
23+
file/metrics:
24+
path: /exported/metrics.json
25+
26+
service:
27+
telemetry:
28+
logs:
29+
level: "debug"
30+
extensions: [basicauth/server]
31+
pipelines:
32+
traces:
33+
receivers: [otlp]
34+
processors: [batch]
35+
exporters: [file/traces, logging]
36+
metrics:
37+
receivers: [otlp]
38+
processors: [batch]
39+
exporters: [file/metrics, logging]

‎services/otel-gateway/test-config.yaml renamed to ‎services/otel-gateway/mock-honeycomb-config.yaml

File renamed without changes.

‎services/otel-gateway/tests.py

+15-5
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,14 @@
1717

1818
import pytest
1919

20-
trace_file = Path("exported/traces.json")
21-
metric_file = Path("exported/metrics.json")
20+
trace_files = [
21+
Path("exported/grafana/traces.json"),
22+
Path("exported/honeycomb/traces.json"),
23+
]
24+
metric_files = [
25+
Path("exported/grafana/metrics.json"),
26+
Path("exported/honeycomb/metrics.json"),
27+
]
2228

2329
# set up trace exporting
2430
tracer_provider = TracerProvider()
@@ -57,7 +63,9 @@ def get_output(path):
5763
time.sleep(0.01)
5864
timeout_count = timeout_count + 1
5965
if timeout_count > 500:
60-
raise Exception("Test timed out - no output written to file after 5 seconds")
66+
raise Exception(
67+
"Test timed out - no output written to file after 5 seconds"
68+
)
6169

6270
return json.loads(path.read_text())
6371

@@ -72,7 +80,8 @@ def service_name_helper(resource_attributes):
7280
)[0]
7381

7482

75-
def test_trace():
83+
@pytest.mark.parametrize("trace_file", trace_files)
84+
def test_trace(trace_file):
7685
generate_test_trace()
7786

7887
output = get_output(trace_file)
@@ -91,7 +100,8 @@ def test_trace():
91100
assert span_data["attributes"][0]["value"] == {"stringValue": "testvalue"}
92101

93102

94-
def test_metric():
103+
@pytest.mark.parametrize("metric_file", metric_files)
104+
def test_metric(metric_file):
95105
generate_test_metric()
96106

97107
output = get_output(metric_file)

0 commit comments

Comments
 (0)
0