8000 Enhance Test and CI Compatibility with openGauss by pangpang20 · Pull Request #10 · HuaweiCloudDeveloper/gaussdb-python · GitHub
[go: up one dir, main page]

Skip to content

Enhance Test and CI Compatibility with openGauss #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Jun 10, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
e99262b
Add try-finally block to ensure proper cursor closure
pangpang20 Jun 6, 2025
f72ab70
Add support for opengauss_skip marker in pytest_collection_modifyitems
pangpang20 Jun 7, 2025
1de855a
Add database type detection and skip tests based on DB type
pangpang20 Jun 7, 2025
9d0ec90
Mark binary copy read tests as unsupported on openGauss and fix line …
pangpang20 Jun 7, 2025
cf0c966
Mark binary copy read tests as unsupported on openGauss and fix line …
pangpang20 Jun 7, 2025
4f0e0a1
Skip nested array test for openGauss
pangpang20 Jun 7, 2025
155a9d0
Skip string encoding tests on openGauss due to binary copy incompatib…
pangpang20 Jun 7, 2025
772f1a0
Skip JSON copy test on openGauss due to binary copy signature mismatch
pangpang20 Jun 7, 2025
2a6c22e
Skip inet/cidr binary copy tests on openGauss due to signature mismatch
pangpang20 Jun 7, 2025
f4b507d
Skip datetime tests on openGauss due to timestamp/year and copy limit…
pangpang20 Jun 7, 2025
689ac22
Skip binary COPY test on openGauss due to incompatible output format
pangpang20 Jun 7, 2025
36d4af2
Skip test_server_version on openGauss due to version format incompati…
pangpang20 Jun 7, 2025
1cf4eb3
Skip test_error_encoding on openGauss due to unsupported DO blocks
pangpang20 Jun 7, 2025
d3a28b4
Skip connection pool tests on openGauss due to lack of backend PID an…
pangpang20 Jun 8, 2025
34e61b2
Skip async pool tests on openGauss due to missing backend PID and pg_…
pangpang20 Jun 8, 2025
815d8a7
Skip broken_reconnect test on openGauss due to missing backend PID su…
pangpang20 Jun 8, 2025
8d80a3a
Skip broken_reconnect async test on openGauss due to missing backend …
pangpang20 Jun 8, 2025
460e5f6
Skip backend pid tests on openGauss
pangpang20 Jun 8, 2025
230e16f
Skip backend pid tests on openGauss
pangpang20 Jun 8, 2025
d0e387e
Skip unsupported SERIALIZABLE level and session timeout on openGauss
pangpang20 Jun 8, 2025
8e64fe2
Skip PGconn.info tests on openGauss
pangpang20 Jun 8, 2025
e883197
Add CI workflow and requirements
pangpang20 Jun 8, 2025
f1eb572
Address code style issues and enhance GaussDB compatibility
pangpang20 Jun 9, 2025
346b001
Switch to Python 3.9 and disable workflow on push events
pangpang20 Jun 10, 2025
3bf61e4
Re-enable lint workflow on push events
pangpang20 Jun 10, 2025
d488b27
Add types-polib to lint workflow dependencies
pangpang20 Jun 10, 2025
0af8c17
Enable async test compatibility and relax timing assertion tolerance
pangpang20 Jun 10, 2025
0d47065
Add Python 3.11 to lint workflow and check sync/async compatibility
pangpang20 Jun 10, 2025
12b8048
Update CI workflow to use fixed database username root
pangpang20 Jun 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,6 @@ per-file-ignores =
# Pytest's importorskip() getting in the way
tests/types/test_numpy.py: E402
tests/types/test_shapely.py: E402

tools/async_to_sync.py: E999
tools/bump_version.py: E999
18 changes: 9 additions & 9 deletions .github/workflows/3rd-party-tests.yml
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
name: 3rd party tests

on:
push:
branches:
- "master"
- "maint-3.1"
- "sqlalchemy_pipeline"
- "django_pipeline"
paths-ignore:
- "docs/*"
- "tools/*"
# push:
# branches:
# - "master"
# - "maint-3.1"
# - "sqlalchemy_pipeline"
# - "django_pipeline"
# paths-ignore:
# - "docs/*"
# - "tools/*"
workflow_dispatch:

concurrency:
Expand Down
104 changes: 104 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
name: CI for gaussdb-python

on:
push:
branches:
- "*"
pull_request:
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref_name }}
cancel-in-progress: true

jobs:
test:
runs-on: ubuntu-22.04

services:
opengauss:
image: opengauss/opengauss-server:latest
ports:
- 5432:5432
env:
GS_USERNAME: root
GS_USER_PASSWORD: ${{ secrets.OPENGAUSS_PASSWORD }}
GS_PASSWORD: ${{ secrets.OPENGAUSS_PASSWORD }}
options: >-
--privileged=true
--name opengauss-custom

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.9"
cache: pip

- name: Create and activate virtual environment
run: |
python -m venv venv
echo "VENV_PATH=$GITHUB_WORKSPACE/venv/bin" >> $GITHUB_ENV
source venv/bin/activate

- name: Install gaussdb libpq driver
run: |
sudo apt update
sudo apt install -y wget unzip
wget -O /tmp/GaussDB_driver.zip https://dbs-download.obs.cn-north-1.myhuaweicloud.com/GaussDB/1730887196055/GaussDB_driver.zip
unzip /tmp/GaussDB_driver.zip -d /tmp/ && rm -rf /tmp/GaussDB_driver.zip
\cp /tmp/GaussDB_driver/Centralized/Hce2_X86_64/GaussDB-Kernel*64bit_Python.tar.gz /tmp/
tar -zxvf /tmp/GaussDB-Kernel*64bit_Python.tar.gz -C /tmp/ && rm -rf /tmp/GaussDB-Kernel*64bit_Python.tar.gz && rm -rf /tmp/psycopg2 && rm -rf /tmp/GaussDB_driver
echo /tmp/lib | sudo tee /etc/ld.so.conf.d/gauss-libpq.conf
sudo sed -i '1s|^|/tmp/lib\n|' /etc/ld.so.conf
sudo ldconfig
ldconfig -p | grep pq

- name: Install dependencies
run: |
source venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -e "./psycopg[dev,test]"
pip install -e ./psycopg_pool


- name: Wait for openGauss to be ready
env:
GSQL_PASSWORD: ${{ secrets.OPENGAUSS_PASSWORD }}
run: |
source venv/bin/activate
for i in {1..15}; do
pg_isready -h localhost -p 5432 -U root && break
sleep 5
done
if ! pg_isready -h localhost -p 5432 -U root; then
echo "openGauss is not ready"
exit 1
fi

- name: Create test database
run: |
docker exec opengauss-custom bash -c "su - omm -c 'gsql -d postgres -c \"CREATE DATABASE test DBCOMPATIBILITY '\''PG'\'';\"'"

- name: Create report directory
run: |
mkdir -p reports

- name: Run tests
env:
PYTHONPATH: ./psycopg:./psycopg_pool
PSYCOPG_IMPL: python
PSYCOPG_TEST_DSN: "host=127.0.0.1 port=5432 dbname=test user=root password=${{ secrets.OPENGAUSS_PASSWORD }} "
run: |
source venv/bin/activate
pytest -s -v

- name: Cleanup
if: always()
run: |
docker stop opengauss-custom
docker rm opengauss-custom
9 changes: 5 additions & 4 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
name: Build documentation

on:
push:
branches:
# This should match the DOC3_BRANCH value in the psycopg-website Makefile
- master
# push:
# branches:
# # This should match the DOC3_BRANCH value in the psycopg-website Makefile
# - master
pull_request:

concurrency:
group: ${{ github.workflow }}-${{ github.ref_name }}
Expand Down
22 changes: 18 additions & 4 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,18 +20,32 @@ jobs:
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
- name: Set up Python 3.9
uses: actions/setup-python@v5
with:
python-version: "3.11"
python-version: "3.9"

- name: install packages to tests
run: pip install ./psycopg[dev,test]
run: |
pip install ./psycopg[dev,test]
pip install types-polib
pip install pre-commit

- name: Lint codebase
run: pre-commit run -a --color=always

- name: Set up Python 3.11
uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install packages for async_to_sync
run: |
pip install ./psycopg[dev,test]
pip install types-polib

- name: Check for sync/async inconsistencies
run: ./tools/async_to_sync.py --check --all
run: ./tools/async_to_sync.py --check $(find tests -name "*_async.py" -type f ! -path "tests/pq/test_async.py")

- name: Install requirements to generate docs
run: sudo apt-get install -y libgeos-dev
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/tests.yml
< F438 td class="blob-code blob-code-addition js-file-line"> # # on.<push|pull_request>.<branches|tags> GitHub Actions docs.
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
name: Tests

on:
push:
# This should disable running the workflow on tags, according to the
# on.<push|pull_request>.<branches|tags> GitHub Actions docs.
branches:
- "*"
# push:
# # This should disable running the workflow on tags, according to the
# branches:
# - "*"
pull_request:
schedule:
- cron: '48 6 * * *'
Expand Down
10 changes: 10 additions & 0 deletions mypy.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[mypy]
python_version = 3.9
ignore_missing_imports = True
check_untyped_defs = True

[mypy-tools.async_to_sync]
ignore_errors = True

[mypy-tools.bump_version]
ignore_errors = True
14 changes: 6 additions & 8 deletions psycopg/psycopg/_copy_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
import sys
import struct
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Any, Generic
from typing import TYPE_CHECKING, Any, Generic, Optional, Tuple
from collections.abc import Sequence

from . import adapt
Expand Down Expand Up @@ -171,7 +171,7 @@ def _read_row_gen(self) -> PQGen[tuple[Any, ...] | None]:

row = self.formatter.parse_row(data)

if row == IS_BINARY_SIGNATURE:
if isinstance(row, str) and row == IS_BINARY_SIGNATURE:
row = yield from self._read_row_gen()

if row is None:
Expand Down Expand Up @@ -267,21 +267,19 @@ def __init__(self, transformer: Transformer):
super().__init__(transformer)
self._signature_sent = False

def parse_row(self, data: Buffer) -> tuple[Any, ...] | str | None:
rv: tuple[Any, ...] | None = None

def parse_row(self, data: Buffer) -> Optional[Tuple[Any, ...]]:
if not self._signature_sent:
if data[: len(_binary_signature)] != _binary_signature:
raise e.DataError(
"binary copy doesn't start with the expected signature"
)
self._signature_sent = True
return IS_BINARY_SIGNATURE
return None

if data != _binary_trailer:
rv = parse_row_binary(data, self.transformer)
return parse_row_binary(data, self.transformer)

return rv
return None

def write(self, buffer: Buffer | str) -> Buffer:
data = self._ensure_bytes(buffer)
Expand Down
4 changes: 3 additions & 1 deletion psycopg/psycopg/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,9 @@ def severity(self) -> str | None:

@property
def severity_nonlocalized(self) -> str | None:
raise NotSupportedError("This is present only in reports generated by libpq versions 9.6 and later.")
raise NotSupportedError(
"This is present only in reports generated by libpq versions 9.6 and later."
)

@property
def sqlstate(self) -> str | None:
Expand Down
6 changes: 2 additions & 4 deletions psycopg/psycopg/types/multirange.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

from .. import _oids
from .. import errors as e
from .. import postgres, sql
from .. import postgres
from ..pq import Format
from ..abc import AdaptContext, Buffer, Dumper, DumperKey, Query
from .range import Range, T, dump_range_binary, dump_range_text, fail_dump
Expand Down Expand Up @@ -47,9 +47,7 @@ def __init__(

@classmethod
def _get_info_query(cls, conn: BaseConnection[Any]) -> Query:
raise e.NotSupportedError(
"multirange types are not supported in GaussDB"
)
raise e.NotSupportedError("multirange types are not supported in GaussDB")

def _added(self, registry: TypesRegistry) -> None:
# Map multiranges ranges and subtypes to info
Expand Down
31 changes: 31 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
anyio==4.9.0
coverage==7.8.2
dnspython==2.7.0
exceptiongroup==1.3.0
gevent==25.5.1
greenlet==3.2.2
idna==3.10
importlib_metadata==8.7.0
iniconfig==2.1.0
Jinja2==3.1.6
MarkupSafe==3.0.2
mypy==1.16.0
mypy_extensions==1.1.0
numpy==2.0.2
packaging==25.0
pathspec==0.12.1
pluggy==1.6.0
pproxy==2.7.9
pytest==8.3.5
pytest-cov==6.1.1
pytest-html==4.1.1
pytest-metadata==3.1.1
pytest-randomly==3.16.0
shapely==2.0.7
sniffio==1.3.1
tomli==2.2.1
typing_extensions==4.13.2
zipp==3.22.0
zope.event==5.0
zope.interface==7.2
types-polib==1.2.0.20250401
Loading
0