8000 Update dag test utils for Airflow 2.0 by leahecole · Pull Request #5865 · GoogleCloudPlatform/python-docs-samples · GitHub
[go: up one dir, main page]

Skip to content

Update dag test utils for Airflow 2.0 #5865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
May 24, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions composer/dag_test_utils/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# CHANGELOG

## 1.0.0 - May 21, 2021
* Upgrade to Airflow 2.0.0
* Add CHANGELOG
* Add README to PyPI
* Add MAINTAINING

## 0.0.1 - May 20, 2021
* Push package compatible with Airflow 1.10.15 to PyPI
38 changes: 38 additions & 0 deletions composer/dag_test_utils/MAINTAINING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Maintenance Guide

## How to release a new version
* Make relevant code changes
* Increment the version number in [setup.py](./setup.py) following [semver](https://semver.org/)
* Add changes to the [CHANGELOG](./CHANGELOG.md)
* If any usage info has changed, update the [README](./README.md)
* [Test the distribution locally](#how-to-test-the-distribution-locally) in the [workflows directory](../workflows)
* Open a PR and request reviews from a [Python Samples owner](https://github.com/orgs/GoogleCloudPlatform/teams/python-samples-owners) and a [Composer Codeowner](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/.github/CODEOWNERS#L24)
* Once the PR is approved and merged, [push the new version to PyPI](#how-to-push-the-new-version-to-pypi)

## How to test the distribution locally
* In a `virtualenv`, run `pip install build`
* Run `python -m build` to build the package. It will create a `dist/` directory containg a wheel (`.whl`) and a tar (`.tar.gz`).
* Change the [`requirements-test.txt`](../workflows/requirements-test.txt) file in the [`workflows`](../workflows) directory to import your distribution relatively, ensuring that `x.y.z` is replaced with your version number

```
../dag_test_utils/dist/cloud_composer_dag_test_utils-x.y.z.tar.gz
```

* Run `nox -s py-3.8 -- quickstart_test.py` (or the entire nox session, if you prefer) - if it passes without an `ImportError`, congrats! You've done it!


## How to push the new version to PyPI
Note - these instructions are derived from [the official guide](https://packaging.python.org/tutorials/packaging-projects/) - if they seem to be out of date, please contact `cloud-dpes-composer@`.

You may only do this after you have successfully tested the package locally and had your PR approved and merged to the primary branch of `python-docs-samples`.

You will need access to the `cloud-dpes-composer` PyPI account and will need an API token. Reach out to `cloud-dpes-composer@` for access.

* In a `virtualenv`, run `pip install build`
* Run `python -m build` to build the package. It will create a `dist/` directory containg a wheel (`.whl`) and a tar (`.tar.gz`).
* Run `pip install twine`
* Run `python -m twine upload --repository pypi dist/*`
* For username, put `__token__`. For password, put your API token.
* Voila! Your new version is fully released!


2 changes: 1 addition & 1 deletion composer/dag_test_utils/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Cloud Composer DAG Testing Utility

This package is used internally to unit test the validity of all Cloud Composer sample DAGs. It is not supported for external production use.
This package is used internally to unit test the validity of all Cloud Composer sample DAGs. It is not supported for external production use. The [latest release can be found on PyPI](https://pypi.org/project/cloud-composer-dag-test-utils/).

## Instructions

Expand Down
3 changes: 2 additions & 1 deletion composer/dag_test_utils/internal_unit_testing.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

# [START composer_dag_unit_testing]
from airflow import models
from airflow.utils.dag_cycle_tester import test_cycle


def assert_has_valid_dag(module):
Expand All @@ -26,7 +27,7 @@ def assert_has_valid_dag(module):
for dag in vars(module).values():
if isinstance(dag, models.DAG):
no_dag_found = False
dag.test_cycle() # Throws if a task cycle is found.
test_cycle(dag) # Throws if a task cycle is found.

if no_dag_found:
raise AssertionError('module does not contain a valid DAG')
Expand Down
13 changes: 11 additions & 2 deletions composer/dag_test_utils/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,22 @@
from setuptools import find_packages
from setuptools import setup

# read the contents of your README file
from os import path
this_directory = path.abspath(path.dirname(__file__))
with open(path.join(this_directory, 'README.md'), encoding='utf-8') as f:
long_description = f.read()


setup(
name="cloud_composer_dag_test_utils",
version="0.0.1",
version="1.0.0",
url="https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/composer/dag_test_utils",
author="Google LLC",
description="Utility used to unit test example Apache Airflow DAGs for Google Cloud Composer. This is not an officially supported Google product.",
long_description=long_description,
long_description_content_type='text/markdown',
packages=find_packages(),
py_modules=['internal_unit_testing'],
install_requires=['apache-airflow[google]==1.10.15']
install_requires=['apache-airflow[google] >= 2.0.0, < 3.0.0']
)
2 changes: 1 addition & 1 deletion composer/workflows/requirements-test.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
pytest==6.2.4
cloud-composer-dag-test-utils==0.0.1
cloud-composer-dag-test-utils==0.0.1
0