8000 Data science onramp Data ingestion by Symmetries · Pull Request #4447 · GoogleCloudPlatform/python-docs-samples · GitHub
[go: up one dir, main page]

Skip to content

Data science onramp Data ingestion #4447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 105 commits into from
Aug 14, 2020

Conversation

Symmetries
Copy link
Contributor

Description

Fixes #

Note: It's a good idea to open an issue first for discussion.

Checklist

Comment on lines 22 to 23
BUCKET_NAME = sys.argv[1]
DATASET_NAME = sys.argv[2]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move args under __main__

Comment on lines 87 to 88
if angle is None:
return None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if not angle:

"--test",
],
"jar_file_uris": [
"gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spacing is too large here. Probably needs to be linted.

Comment on lines 78 to 79
operation = cluster_client.create_cluster(PROJECT, REGION, CLUSTER_DATA)

10000 Copy link
Collaborator
@bradmiro bradmiro Aug 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Monday a new version of the python-dataproc library was released with breaking changes. project_id, region and cluster are now key word args. Can you please update accordingly? I'm happy to help with this also.

Comment on lines 86 to 87
operation = cluster_client.delete_cluster(PROJECT, REGION,
CLUSTER_NAME)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to project_id=PROJECT, region=REGION, cluster_name=CLUSTER_NAME please.

Comment on lines 147 to 148
response = job_client.submit_job_as_operation(project_id=PROJECT, region=REGION,
job=JOB_DETAILS)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks correct with respect to API update.

google-auth-httplib2==0.0.3
google-cloud==0.34.0
google-cloud-storage==1.28.1
google-cloud-dataproc==0.8.0
Copy link
Collaborator
@bradmiro bradmiro Aug 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

google-cloud-dataproc==2.0.0

@google-cla
Copy link
google-cla bot commented Aug 12, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link
google-cla bot commented Aug 13, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link
google-cla bot commented Aug 13, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@leahecole leahecole added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Aug 13, 2020
@leahecole leahecole changed the title WIP: Data science onramp Data ingestion Data science onramp Data ingestion Aug 13, 2020
@google-cla
Copy link
google-cla bot commented Aug 13, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Aug 13, 2020
@google-cla
Copy link
google-cla bot commented Aug 13, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Aug 13, 2020
@leahecole leahecole added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 13, 2020
@leahecole
Copy link
Collaborator

gonna rerun to see if it's a flake and there were other tests running

@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 13, 2020
@leahecole leahecole merged commit ca4e6a4 into GoogleCloudPlatform:master Aug 14, 2020
@tmatsuo
Copy link
Contributor
tmatsuo commented Aug 14, 2020

@leahecole @bradmiro This PR deletes the top level .gitignore. Please add it back.

@tmatsuo
Copy link
Contributor
tmatsuo commented Aug 15, 2020

nm, #4502

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants
0