8000 storage: S3 SDK sample to list buckets by frankyn · Pull Request #2022 · GoogleCloudPlatform/python-docs-samples · GitHub
[go: up one dir, main page]

Skip to content

storage: S3 SDK sample to list buckets #2022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Mar 7, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions storage/s3-sdk/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
.. This file is automatically generated. Do not edit this file directly.

Google Cloud Storage Python Samples
===============================================================================

.. image:: https://gstatic.com/cloudssh/images/open-btn.png
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=storage/s3-sdk/README.rst


This directory contains samples for Google Cloud Storage. `Google Cloud Storage`_ provides support for S3 API users through the GCS XML API.
Learn more about migrating data from S3 to GCS at https://cloud.google.com/storage/docs/migrating.




.. _Google Cloud Storage: https://cloud.google.com/storage/docs

Setup
-------------------------------------------------------------------------------


Install Dependencies
++++++++++++++++++++

#. Clone python-docs-samples and change directory to the sample directory you want to use.

.. code-block:: bash

$ git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest you add the cd command after the clone.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, should this tell the user to set the authentication environment variables?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I modified the example to pass in credentials using CLI parameter.

I looked at other samples and I didn't see cd ./../ when generating README's with templates. For consistency, I'm going to keep as is as it impacts other samples.


#. Install `pip`_ and `virtualenv`_ if you do not already have them. You may want to refer to the `Python Development Environment Setup Guide`_ for Google Cloud Platform for instructions.

.. _Python Development Environment Setup Guide:
https://cloud.google.com/python/setup

#. Create a virtualenv. Samples are compatible with Python 2.7 and 3.4+.

.. code-block:: bash

$ virtualenv env
$ source env/bin/activate

#. Install the dependencies needed to run the samples.

.. code-block:: bash

$ pip install -r requirements.txt

.. _pip: https://pip.pypa.io/
.. _virtualenv: https://virtualenv.pypa.io/

Samples
-------------------------------------------------------------------------------

List GCS Buckets using S3 SDK
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

.. image:: https://gstatic.com/cloudssh/images/open-btn.png
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=storage/s3-sdk/list_gcs_buckets.py,storage/s3-sdk/README.rst




To run this sample:

.. code-block:: bash

$ python list_gcs_buckets.py

usage: list_gcs_buckets.py [-h] google_access_key_id google_access_key_secret

positional arguments:
google_access_key_id Your Cloud Storage HMAC Access Key ID.
google_access_key_secret
Your Cloud Storage HMAC Access Key Secret.

optional arguments:
-h, --help show this help message and exit





.. _Google Cloud SDK: https://cloud.google.com/sdk/
23 changes: 23 additions & 0 deletions storage/s3-sdk/README.rst.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# This file is used to generate README.rst

product:
name: Google Cloud Storage
short_name: Cloud Storage
url: https://cloud.google.com/storage/docs
description: >
`Google Cloud Storage`_ provides support for S3 API users through the
GCS XML API.

Learn more about migrating data from S3 to GCS at https://cloud.google.com/storage/docs/migrating.

setup:
- install_deps

samples:
- name: List GCS Buckets using S3 SDK
file: list_gcs_buckets.py
show_help: true

cloud_client_library: false

folder: storage/s3-sdk
55 changes: 55 additions & 0 deletions storage/s3-sdk/list_gcs_buckets.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#!/usr/bin/env python

# Copyright 2019 Google, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import argparse
# [START storage_s3_sdk_list_buckets]
import boto3


def list_gcs_buckets(google_access_key_id, google_access_key_secret):
"""Lists all GCS buckets using boto3 SDK"""
# Create a new client and do the following:
# 1. Change the endpoint URL to use the
# Google Cloud Storage XML API endpoint.
# 2. Use Cloud Storage HMAC Credentials.
client = boto3.client("s3", region_name="auto",
endpoint_url="https://storage.googleapis.com",
aws_access_key_id=google_access_key_id,
aws_secret_access_key=google_access_key_secret)

# Call GCS to list current buckets
response = client.list_buckets()

# Print bucket names
print("Buckets:")
for bucket in response["Buckets"]:
print(bucket["Name"])
# [END storage_s3_sdk_list_buckets]


if __name__ == "__main__":
parser = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("google_access_key_id",
help="Your Cloud Storage HMAC Access Key ID.")
parser.add_argument("google_access_key_secret",
help="Your Cloud Storage HMAC Access Key Secret.")

args = parser.parse_args()

list_gcs_buckets(google_access_key_id=args.google_access_key_id,
google_access_key_secret=args.google_access_key_secret)
28 changes: 28 additions & 0 deletions storage/s3-sdk/list_gcs_buckets_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Copyright 2019 Google, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import os

import list_gcs_buckets

BUCKET = os.environ["GOOGLE_CLOUD_PROJECT_S3_SDK"]
KEY_ID = os.environ["STORAGE_HMAC_ACCESS_KEY_ID"]
SECRET_KEY = os.environ["STORAGE_HMAC_ACCESS_SECRET_KEY"]


def test_list_blobs(capsys):
list_gcs_buckets.list_gcs_buckets(google_access_key_id=KEY_ID,
google_access_key_secret=SECRET_KEY)
out, _ = capsys.readouterr()
assert BUCKET in out
1 change: 1 addition & 0 deletions storage/s3-sdk/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
boto3==1.9.38
Binary file modified testing/secrets.tar.enc
Binary file not shown.
0