8000 storage: S3 SDK sample to list buckets (#2022) · mmmarklu/python-docs-samples@6ea559f · GitHub
[go: up one dir, main page]
Skip to content

Commit 6ea559f

Browse files
frankynengelke
frankyn
authored and
engelke
committed
storage: S3 SDK sample to list buckets (GoogleCloudPlatform#2022)
* Add canonical boto3 sample for review. * Update path, add README, and add system test to sample. * Add newline * Fix wording * Fix typos * Fix region tag names * Update variable name for interop client * Update README to not include Google Auth * Fix quotes and update environment variable * Address feedback * Update readme * Add HMAC credentials to secrets.tar.enc * Fix lint issue
1 parent 4e994b0 commit 6ea559f

File tree

6 files changed

+191
-0
lines changed
  • list_gcs_buckets.py
  • list_gcs_buckets_test.py
  • requirements.txt
  • testing

    • 6 files changed

    +191
    -0
    lines changed

    ‎storage/s3-sdk/README.rst

    Lines changed: 84 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1,84 @@
    1+
    .. This file is automatically generated. Do not edit this file directly.
    2+
    3+
    Google Cloud Storage Python Samples
    4+
    ===============================================================================
    5+
    6+
    .. image:: https://gstatic.com/cloudssh/images/open-btn.png
    7+
    :target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=storage/s3-sdk/README.rst
    8+
    9+
    10+
    This directory contains samples for Google Cloud Storage. `Google Cloud Storage`_ provides support for S3 API users through the GCS XML API.
    11+
    Learn more about migrating data from S3 to GCS at https://cloud.google.com/storage/docs/migrating.
    12+
    13+
    14+
    15+
    16+
    .. _Google Cloud Storage: https://cloud.google.com/storage/docs
    17+
    18+
    Setup
    19+
    -------------------------------------------------------------------------------
    20+
    21+
    22+
    Install Dependencies
    23+
    ++++++++++++++++++++
    24+
    25+
    #. Clone python-docs-samples and change directory to the sample directory you want to use.
    26+
    27+
    .. code-block:: bash
    28+
    29+
    $ git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
    30+
    31+
    #. Install `pip`_ and `virtualenv`_ if you do not already have them. You may want to refer to the `Python Development Environment Setup Guide`_ for Google Cloud Platform for instructions.
    32+
    33+
    .. _Python Development Environment Setup Guide:
    34+
    https://cloud.google.com/python/setup
    35+
    36+
    #. Create a virtualenv. Samples are compatible with Python 2.7 and 3.4+.
    37+
    38+
    .. code-block:: bash
    39+
    40+
    $ virtualenv env
    41+
    $ source env/bin/activate
    42+
    43+
    #. Install the dependencies needed to run the samples.
    44+
    45+
    .. code-block:: bash
    46+
    47+
    $ pip install -r requirements.txt
    48+
    49+
    .. _pip: https://pip.pypa.io/
    50+
    .. _virtualenv: https://virtualenv.pypa.io/
    51+
    52+
    Samples
    53+
    -------------------------------------------------------------------------------
    54+
    55+
    List GCS Buckets using S3 SDK
    56+
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    57+
    58+
    .. image:: https://gstatic.com/cloudssh/images/open-btn.png
    59+
    :target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=storage/s3-sdk/list_gcs_buckets.py,storage/s3-sdk/README.rst
    60+
    61+
    62+
    63+
    64+
    To run this sample:
    65+
    66+
    .. code-block:: bash
    67+
    68+
    $ python list_gcs_buckets.py
    69+
    70+
    usage: list_gcs_buckets.py [-h] google_access_key_id google_access_key_secret
    71+
    72+
    positional arguments:
    73+
    google_access_key_id Your Cloud Storage HMAC Access Key ID.
    74+
    google_access_key_secret
    75+
    Your Cloud Storage HMAC Access Key Secret.
    76+
    77+
    optional arguments:
    78+
    -h, --help show this help message and exit
    79+
    80+
    81+
    82+
    83+
    84+
    .. _Google Cloud SDK: https://cloud.google.com/sdk/

    ‎storage/s3-sdk/README.rst.in

    Lines changed: 23 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1,23 @@
    1+
    # This file is used to generate README.rst
    2+
    3+
    product:
    4+
    name: Google Cloud Storage
    5+
    short_name: Cloud Storage
    6+
    url: https://cloud.google.com/storage/docs
    7+
    description: >
    8+
    `Google Cloud Storage`_ provides support for S3 API users through the
    9+
    GCS XML API.
    10+
    11+
    Learn more about migrating data from S3 to GCS at https://cloud.google.com/storage/docs/migrating.
    12+
    13+
    setup:
    14+
    - install_deps
    15+
    16+
    samples:
    17+
    - name: List GCS Buckets using S3 SDK
    18+
    file: list_gcs_buckets.py
    19+
    show_help: true
    20+
    21+
    cloud_client_library: false
    22+
    23+
    folder: storage/s3-sdk

    ‎storage/s3-sdk/list_gcs_buckets.py

    Lines changed: 55 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1,55 @@
    1+
    #!/usr/bin/env python
    2+
    3+
    # Copyright 2019 Google, Inc.
    4+
    #
    5+
    # Licensed under the Apache License, Version 2.0 (the "License");
    6+
    # you may not use this file except in compliance with the License.
    7+
    # You may obtain a copy of the License at
    8+
    #
    9+
    # http://www.apache.org/licenses/LICENSE-2.0
    10+
    #
    11+
    # Unless required by applicable law or agreed to in writing, software
    12+
    # distributed under the License is distributed on an "AS IS" BASIS,
    13+
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    14+
    # See the License for the specific language governing permissions and
    15+
    # limitations under the License.
    16+
    17+
    import argparse
    18+
    # [START storage_s3_sdk_list_buckets]
    19+
    import boto3
    20+
    21+
    22+
    def list_gcs_buckets(google_access_key_id, google_access_key_secret):
    23+
    """Lists all GCS buckets using boto3 SDK"""
    24+
    # Create a new client and do the following:
    25+
    # 1. Change the endpoint URL to use the
    26+
    # Google Cloud Storage XML API endpoint.
    27+
    # 2. Use Cloud Storage HMAC Credentials.
    28+
    client = boto3.client("s3", region_name="auto",
    29+
    endpoint_url="https://storage.googleapis.com",
    30+
    aws_access_key_id=google_access_key_id,
    31+
    aws_secret_access_key=google_access_key_secret)
    32+
    33+
    # Call GCS to list current buckets
    34+
    response = client.list_buckets()
    35+
    36+
    # Print bucket names
    37+
    print("Buckets:")
    38+
    for bucket in response["Buckets"]:
    39+
    print(bucket["Name"])
    40+
    # [END storage_s3_sdk_list_buckets]
    41+
    42+
    43+
    if __name__ == "__main__":
    44+
    parser = argparse.ArgumentParser(
    45+
    description=__doc__,
    46+
    formatter_class=argparse.RawDescriptionHelpFormatter)
    47+
    parser.add_argument("google_access_key_id",
    48+
    help="Your Cloud Storage HMAC Access Key ID.")
    49+
    parser.add_argument("google_access_key_secret",
    50+
    help="Your Cloud Storage HMAC Access Key Secret.")
    51+
    52+
    args = parser.parse_args()
    53+
    54+
    list_gcs_buckets(google_access_key_id=args.google_access_key_id,
    55+
    google_access_key_secret=args.google_access_key_secret)

    ‎storage/s3-sdk/list_gcs_buckets_test.py

    Lines changed: 28 additions & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1,28 @@
    1+
    # Copyright 2019 Google, Inc.
    2+
    #
    3+
    # Licensed under the Apache License, Version 2.0 (the "License");
    4+
    # you may not use this file except in compliance with the License.
    5+
    # You may obtain a copy of the License at
    6+
    #
    7+
    # http://www.apache.org/licenses/LICENSE-2.0
    8+
    #
    9+
    # Unless required by applicable law or agreed to in writing, software
    10+
    # distributed under the License is distributed on an "AS IS" BASIS,
    11+
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    12+
    # See the License for the specific language governing permissions and
    13+
    # limitations under the License.
    14+
    15+
    import os
    16+
    17+
    import list_gcs_buckets
    18+
    19+
    BUCKET = os.environ["GOOGLE_CLOUD_PROJECT_S3_SDK"]
    20+
    KEY_ID = os.environ["STORAGE_HMAC_ACCESS_KEY_ID"]
    21+
    SECRET_KEY = os.environ["STORAGE_HMAC_ACCESS_SECRET_KEY"]
    22+
    23+
    24+
    def test_list_blobs(capsys):
    25+
    list_gcs_buckets.list_gcs_buckets(google_access_key_id=KEY_ID,
    26+
    google_access_key_secret=SECRET_KEY)
    27+
    out, _ = capsys.readouterr()
    28+
    assert BUCKET in out

    ‎storage/s3-sdk/requirements.txt

    Lines changed: 1 addition & 0 deletions
    Original file line numberDiff line numberDiff line change
    @@ -0,0 +1 @@
    1+
    boto3==1.9.38

    ‎testing/secrets.tar.enc

    0 Bytes
    Binary file not shown.

    0 commit comments

    Comments
     (0)
    0