-
Notifications
You must be signed in to change notification settings - Fork 6.5k
Data science onramp Data ingestion #4447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data science onramp Data ingestion #4447
Conversation
This reverts commit 580c8e1.
…ython-docs-samples into feature-engineering
BUCKET_NAME = sys.argv[1] | ||
DATASET_NAME = sys.argv[2] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move args under __main__
if angle is None: | ||
return None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: if not angle:
"--test", | ||
], | ||
"jar_file_uris": [ | ||
"gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Spacing is too large here. Probably needs to be linted.
operation = cluster_client.create_cluster(PROJECT, REGION, CLUSTER_DATA) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On Monday a new version of the python-dataproc library was released with breaking changes. project_id
, region
and cluster
are now key word args. Can you please update accordingly? I'm happy to help with this also.
operation = cluster_client.delete_cluster(PROJECT, REGION, | ||
CLUSTER_NAME) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change to project_id=PROJECT, region=REGION, cluster_name=CLUSTER_NAME
please.
response = job_client.submit_job_as_operation(project_id=PROJECT, region=REGION, | ||
job=JOB_DETAILS) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks correct with respect to API update.
google-auth-httplib2==0.0.3 | ||
google-cloud==0.34.0 | ||
google-cloud-storage==1.28.1 | ||
google-cloud-dataproc==0.8.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
google-cloud-dataproc==2.0.0
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
743a664
to
b82059b
Compare
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
gonna rerun to see if it's a flake and there were other tests running |
@leahecole @bradmiro This PR deletes the top level |
nm, #4502 |
Description
Fixes #
Note: It's a good idea to open an issue first for discussion.
Checklist
nox -s py-3.6
(see Test Environment Setup)nox -s lint
(see Test Environment Setup)