8000 cleanup: Upgrade Composer samples to Airflow 2.0 by leahecole · Pull Request #5782 · GoogleCloudPlatform/python-docs-samples · GitHub
[go: up one dir, main page]
Skip to content

cleanup: Upgrade Composer samples to Airflow 2.0 #5782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 31 commits into from
May 27, 2021
Merged

cleanup: Upgrade Composer samples to Airflow 2.0 #5782

merged 31 commits into from
May 27, 2021

Conversation

leahecole
Copy link
Collaborator
@leahecole leahecole commented May 3, 2021 •
edited
Loading

Description

This PR follows best practices for upgrading DAGs from Airflow 1 to Airflow 2. Changes included:

  • Adding an additional conftest.py to the blogs folder. For now, that DAG will stay at using Airflow 1.0, and we need to initialize a compatible DB for those tests separate from the workflows DB.
  • Use of the newly released unit test util in workflows/requirements-test.txt
  • Imports changed from contrib to providers
  • Removal of samples that showcased running Python 2 code
  • Referring to the officially supported Airflow constraints file instead of the piecemeal one I had
  • Making the appropriate changes on anything previously throwing a Deprecation Warning (i.e. bash_operator -> bash)

If you would like access to my Composer environment to see the DAGs deployed, please let me know. Otherwise, here is a screenshot showing successful dag runs across the board

Fixes #5171
DataflowTemplateOperator tutorial update in CL/371792521 - removed instructions for setting gce_region Airflow variable (@sofisl )

CHANGE NOT ADDRESSED - I need to look further at the DAG example we have triggered by GCF. That will be addressed in an upcoming PR.

Note: It's a good idea to open an issue first for discussion.

Checklist

@product-auto-label product-auto-label bot added the samples Issues that are directly related to samples. label May 3, 2021
@snippet-bot
Copy link
snippet-bot bot commented May 3, 2021 •
edited
Loading
You are about to delete the following sample browser pages.

Here is the summary of changes.

You are about to add 1 region tag.
You are about to delete 3 region tags.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label May 3, 2021
@leahecole leahecole mentioned this pull request May 7, 2021
Merged
8 tasks
@leahecole leahecole force-pushed the airflow_migration branch from 78a9ae2 to 7c60e1c Compare May 14, 2021 22:31
@leahecole
Copy link
Collaborator Author

YAY #5806 made the tests pass!! This is only red because I'm bad at remembering to run lint locally!!!!!!!!!!!!!

@leahecole leahecole mentioned this pull request May 19, 2021
Merged
8 tasks
@leahecole leahecole added the status: blocked Resolving the issue is dependent on other work. label May 21, 2021
@leahecole leahecole mentioned this pull request May 24, 2021
Merged
8 tasks
@leahecole leahecole force-pushed the airflow_migration branch from 05cb6f9 to f87bcac Compare May 24, 2021 21:11
@leahecole leahecole marked this pull request as ready for review May 24, 2021 21:44
@leahecole leahecole requested review from rachael-ds and a team as code owners May 24, 2021 21:44
@leahecole leahecole changed the title [WIP] Upgrade Composer samples to Airflow 2.0 cleanup: Upgrade Composer samples to Airflow 2.0 May 24, 2021
@leahecole leahecole removed the status: blocked Resolving the issue is dependent on other work. label May 24, 2021
@leahecole leahecole requested a review from sofisl May 24, 2021 21:58
@sofisl
Copy link
Contributor
sofisl commented May 24, 2021

Just looked at the dataflow/dataproc tutorials, LGTM!

rachael-ds
rachael-ds reviewed May 26, 2021
View reviewed changes
@leahecole leahecole requested a review from rachael-ds May 26, 2021 18:01
@leahecole leahecole mentioned this pull request May 26, 2021
Closed
1 task
rachael-ds
rachael-ds approved these changes May 27, 2021
View reviewed changes
Copy link
Contributor
@rachael-ds rachael-ds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@leahecole
Copy link
Collaborator Author

omg I have no idea how I made this mess with adding these requirements files. I'll fix later today 🤦🏻‍♀️

@leahecole leahecole force-pushed the airflow_migration branch 2 times, most recently from affa66f to 207371f Compare May 27, 2021 16:18
@leahecole leahecole force-pushed the airflow_migration branch from 207371f to f9447fa Compare May 27, 2021 16:19
@leahecole leahecole added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label May 27, 2021
@leahecole
Copy link
Collaborator Author

I have been talking to TWs and need to move the Airflow 1 versions to a "legacy samples" folder

@leahecole leahecole removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label May 27, 2021
@leahecole
Copy link
Collaborator Author

Actually, I'll do the legacy samples thing in a separate PR. I'm ready for you, @dinagraves ! :)

@leahecole leahecole mentioned this pull request May 27, 2021
Closed
dinagraves
dinagraves approved these changes May 27, 2021
View reviewed changes
@leahecole leahecole merged commit 675e55f into master May 27, 2021
@leahecole leahecole deleted the airflow_migration branch May 27, 2021 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rachael-ds rachael-ds rachael-ds approved these changes

@dinagraves dinagraves dinagraves approved these changes

@sofisl sofisl sofisl approved these changes

Assignees

@dinagraves dinagraves

Labels
cla: yes This human has signed the Contributor License Agreement. samples Issues that are directly related to samples.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update composer workflow samples
4 participants
@leahecole @sofisl @rachael-ds @dinagraves
0