8000 Sam/sync versions (#925) · cepro/postgres@c506dff · GitHub
[go: up one dir, main page]
Skip to content

Commit c506dff

Browse files
samrosesamrose
authored
Sam/sync versions (supabase#925)
* feat: integrating a parallel nix-based build of supabase psql, exts, wrappers includes experimental integration of orioledb and patched psql 16 for orioledb * docs: updating docs to correspond with new location * feat: clearing out commented out citus references in the nix implementation * chore: fmt * chore: update maintainer gh username * feat: update pgvector to 0.6.0 * feat: include nix specific ci (may require env vars configured in repo settings) * fix: deprecate justfile * feat: wip update versions from ansible vars data * chore: stashing progress * feat: WIP tooling that will update versions, source and cargovendor sha * feat: more work toward versions sync * fix: gh action targets * fix: conform name * feat: bring all exts into local/native build (no more upstream) run the sync util on the nix expressions, and then build and test against those versions as synced from ansible/vars.yml paring down to psql 15.6 + extensions/wrappers (put aside oriole and psql 16 for now, will bring back when ready)
1 parent b7651a6 commit c506dff

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

70 files changed

+4568
-0
lines changed

‎.github/workflows/nix-build.yml

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name: Nix CI
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
9+
permissions: read-all
10+
11+
jobs:
12+
build:
13+
strategy:
14+
fail-fast: false
15+
matrix:
16+
os: [ ubuntu-latest ]
17+
cmd: [ "nix flake check -L --show-trace", "nix build .#psql_15/bin .#psql_15/docker"]
18+
runs-on: ${{ matrix.os }}
19+
name: nix-build
20+
steps:
21+
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9
22+
with:
23+
fetch-depth: 0
24+
- uses: DeterminateSystems/nix-installer-action@65d7c888b2778e8cf30a07a88422ccb23499bfb8
25+
- uses: DeterminateSystems/magic-nix-cache-action@749fc5bbc9fa49d60c2b93f6c4bc867b82e1d295
26+
- run: ${{ matrix.cmd }}

‎.github/workflows/nix-cache-upload.yml

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Nix Cache upload
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
8+
permissions:
9+
contents: write
10+
packages: write
11+
id-token: write
12+
13+
jobs:
14+
build:
15+
strategy:
16+
fail-fast: false
17+
runs-on: ubuntu-latest
18+
name: nix-build
19+
steps:
20+
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9
21+
with:
22+
fetch-depth: 0
23+
- uses: DeterminateSystems/nix-installer-action@65d7c888b2778e8cf30a07a88422ccb23499bfb8
24+
- uses: DeterminateSystems/magic-nix-cache-action@749fc5bbc9fa49d60c2b93f6c4bc867b82e1d295
25+
- name: configure aws credentials for s3
26+
uses: aws-actions/configure-aws-credentials@v1
27+
with:
28+
role-to-assume: ${{ secrets.DEV_AWS_ROLE }}
29+
aws-region: "us-east-1"
30+
kvm: true
31+
extra-conf: |
32+
system-features = kvm
33+
34+
- name: write secret key
35+
# use python so we don't interpolate the secret into the workflow logs, in case of bugs
36+
run: |
37+
python -c "import os; file = open('nix-secret-key', 'w'); file.write(os.environ['NIX_SIGN_SECRET_KEY']); file.close()"
38+
env:
39+
NIX_SIGN_SECRET_KEY: ${{ secrets.NIX_SIGN_SECRET_KEY }}
40+
41+
- name: build and copy to S3
42+
run: |
43+
for x in 15 16 orioledb_16; do
44+
nix build .#psql_$x/bin -o result-$x
45+
done
46+
nix copy --to s3://nix-postgres-artifacts?secret-key=nix-secret-key ./result*

‎.github/workflows/nix-docker-image.yml

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
name: Nix Docker
2+
3+
on:
4+
push:
5+
branches: [ 'main' ]
6+
7+
env:
8+
REGISTRY: ghcr.io
9+
IMAGE_NAME: ${{ github.repository }}
10+
11+
jobs:
12+
build:
13+
name: "update: build and deploy postgres server images"
14+
runs-on: ubuntu-latest
15+
permissions:
16+
contents: write
17+
packages: write
18+
steps:
19+
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9
20+
with:
21+
fetch-depth: 0
22+
- uses: DeterminateSystems/nix-installer-action@main
23+
- uses: DeterminateSystems/magic-nix-cache-action@main
24+
- uses: actions/checkout@v3
25+
26+
- name: Build images
27+
run: |
28+
nix build .#psql_15/docker -o result-docker-15
29+
- name: Log in to the Container registry
30+
uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a # v2.1.0
31+
with:
32+
registry: ${{ env.REGISTRY }}
33+
username: ${{ github.actor }}
34+
password: ${{ secrets.GITHUB_TOKEN }}
35+
36+
- name: Push image to the Container registry
37+
run: |
38+
set -x
39+
for x in 15; do
40+
nix build .#psql_$x/docker.copyToRegistry
41+
done

‎.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,6 @@ ansible/image-manifest*.json
1010
__pycache__/
1111
*.py[cod]
1212
*$py.class
13+
14+
#nix related
15+
result*

‎README.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,3 +109,12 @@ $ time packer build -timestamp-ui \
109109
We are building the features of Firebase using enterprise-grade, open source products. We support existing communities wherever possible, and if the products don’t exist we build them and open source them ourselves.
110110

111111
[![New Sponsor](https://user-images.githubusercontent.com/10214025/90518111-e74bbb00-e198-11ea-8f88-c9e3c1aa4b5b.png)](https://github.com/sponsors/supabase)
112+
113+
114+
## Experimental Nix Packaging of resources
115+
116+
There is a `/nix` folder in this repo, plus a `flake.nix` and `flake.lock` that facilitate using the Nix package management system to package supabase/postgres, and all of our extensions and wrappers. A user will need nix installed on their machine. As of 4/1/2024 the package set only builds on target machines (`x86_64-linux` and `aarch64-linux`), however work is under way to also support building and using directly on `aarch64-darwin` (macOs). As of 4/1/2024, versions of packages and extensions are synced from `/ansible/vars.yml` via a utility that can be run by executing `nix run .#sync-exts-versions` (you must have nix installed and be on the supported `x86_64-linux` and `aarch64-linux` for this command to work). The short term goal is to sync these versions as they are updated by our infrastructure and postgres teams, then to see the nix packaged versions build successfully in parallel over time, along with tests of the nix packaged versions passing.
117+
118+
The supabase/postgres repo will continue to source it's dependencies from ansible for the short term, while we stabilize this nix build.
119+
120+
Forthcoming PR's will include: integrating the nix work into our ansible/packer builds, building natively on aarch64-darwin (macOs), more testing

‎flake.lock

Lines changed: 180 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)
0