8000 Add GCF Python SQL samples by ace-n · Pull Request #1655 · GoogleCloudPlatform/python-docs-samples · GitHub
[go: up one dir, main page]

Skip to content

Add GCF Python SQL samples #1655

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Sep 12, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .kokoro/presubmit_tests_functions_sql.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Format: //devtools/kokoro/config/proto/build.proto

# Download secrets from Cloud Storage.
gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/python-docs-samples"

# Tell the trampoline which build file to use.
env_vars: {
key: "TRAMPOLINE_BUILD_FILE"
value: "github/python-docs-samples/.kokoro/system_tests.sh"
}

env_vars: {
key: "NOX_SESSION"
value: "functions and sql and py36 and not venv"
}
10 changes: 10 additions & 0 deletions .kokoro/system_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,19 @@ source ./testing/test-env.sh
export GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/testing/service-account.json
export GOOGLE_CLIENT_SECRETS=$(pwd)/testing/client-secrets.json

# Run Cloud SQL proxy, if required
if [ -n "${CLOUD_SQL_PROXY}" ]; then
cloud_sql_proxy -instances="${MYSQL_INSTANCE}"=tcp:3306 &
cloud_sql_proxy -instances="${POSTGRES_INSTANCE}"=tcp:5432 &
fi

# Run tests
nox -k "${NOX_SESSION}" || ret_code=$?

if [ -n "${CLOUD_SQL_PROXY}" ]; then
killall cloud_sql_proxy || true
fi

# Workaround for Kokoro permissions issue: delete secrets
rm testing/{test-env.sh,client-secrets.json,service-account.json}

Expand Down
68 changes: 68 additions & 0 deletions functions/sql/mysql_sample.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# [START functions_sql_mysql]
from os import getenv

import pymysql

is_production = getenv('SUPERVISOR_HOSTNAME') is not None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any more straightforward methods of detecting the environment? Even better, could we do try/except to simply try to connect to the unix socket and fallback to user settings if it doesn't work? (properly commented)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We ideally should not use SUPERVISOR_HOSTNAME. It may not be there in the future. You opt for a separate env var to contain the "key". So.. INSTANCE_CONNECTION_TYPE with either unix_socket (or empty string for default), of host for traditional connections.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Node samples use the NODE_ENV env var to determine whether a function has been deployed. For the sake of maintaining patterns across samples, is there a way we can do this here?


# TODO(developer): specify SQL connection details
CONNECTION_NAME = getenv(
'INSTANCE_CONNECTION_NAME',
'<YOUR INSTANCE CONNECTION NAME>')
DB_USER = getenv('MYSQL_USER', '<YOUR DB USER>')
DB_PASSWORD = getenv('MYSQL_PASSWORD', '<YOUR DB PASSWORD>')
DB_NAME = getenv('MYSQL_DATABASE', '<YOUR DB NAME>')

mysql_config = {
'user': DB_USER,
'password': DB_PASSWORD,
'db': DB_NAME,
'charset': 'utf8mb4',
'cursorclass': pymysql.cursors.DictCursor,
'autocommit': True
}

if is_production:
mysql_config['unix_socket'] = \
'/cloudsql/' + CONNECTION_NAME

# Create SQL connection globally to enable reuse
# PyMySQL does not include support for connection pooling
mysql_conn = pymysql.connect(**mysql_config)


def __get_cursor():
"""
Helper function to get a cursor
PyMySQL does NOT automatically reconnect,
so we must reconnect explicitly using ping()
"""
try:
return mysql_conn.cursor()
except Exception:
mysql_conn.ping(reconnect=True)
return mysql_conn.cursor()


def mysql_demo(request):
# Remember to close SQL resources declared while running this function.
# Keep any declared in global scope (e.g. mysql_conn) for later reuse.
with __get_cursor() as cursor:
cursor.execute('SELECT NOW() as now')
results = cursor.fetchone()
return str(results['now'])
# [END functions_sql_mysql]
19 changes: 19 additions & 0 deletions functions/sql/mysql_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import mysql_sample


def test_mysql():
mysql_sample.mysql_demo(None)
61 changes: 61 additions & 0 deletions functions/sql/postgres_sample.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# [START functions_sql_postgres]
from os import getenv

from psycopg2.pool import SimpleConnectionPool

is_production = getenv('SUPERVISOR_HOSTNAME') is not None

# TODO(developer): specify SQL connection details
CONNECTION_NAME = getenv(
'INSTANCE_CONNECTION_NAME',
'<YOUR INSTANCE CONNECTION NAME>')
DB_USER = getenv('POSTGRES_USER', '<YOUR DB USER>')
DB_PASSWORD = getenv('POSTGRES_PASSWORD', '<YOUR DB PASSWORD>')
DB_NAME = getenv('POSTGRES_DATABASE', '<YOUR DB NAME>')

pg_config = {
'user': DB_USER,
'password': DB_PASSWORD,
'dbname': DB_NAME,
}

if is_production:
pg_config['host'] = '/cloudsql/' + CONNECTION_NAME
else:
pg_config['host'] = 'localhost'

# Connection pools reuse connections between invocations,
# and handle dropped or expired connections automatically.
pg_pool = None


def postgres_demo(request):
global pg_pool

# Initialize the pool lazily, in case SQL access isn't needed for this
# GCF instance. Doing so minimizes the number of active SQL connections,
# which helps keep your GCF instances under SQL connection limits.
if not pg_pool:
pg_pool = SimpleConnectionPool(1, 1, **pg_config)

# Remember to close SQL resources declared while running this function.
# Keep any declared in global scope (e.g. pg_pool) for later reuse.
with pg_pool.getconn().cursor() as cursor:
cursor.execute('SELECT NOW() as now')
results = cursor.fetchone()
return str(results[0])
# [END functions_sql_postgres]
19 changes: 19 additions & 0 deletions functions/sql/postgres_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import postgres_sample


def test_postgres():
postgres_sample.postgres_demo(None)
2 changes: 2 additions & 0 deletions functions/sql/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
psycopg2==2.7.5
PyMySQL==0.9.2
Binary file modified testing/secrets.tar.enc
Binary file not shown.
12 changes: 12 additions & 0 deletions testing/test-env.tmpl.sh
Original file line number Diff line number Diff line change
@@ -1,13 +1,25 @@
# Environment variables for system tests.
export GCLOUD_PROJECT=your-project-id
export GOOGLE_CLOUD_PROJECT=$GCLOUD_PROJECT
export FIRESTORE_PROJECT=

export CLOUD_STORAGE_BUCKET=$GCLOUD_PROJECT
export API_KEY=
export BIGTABLE_CLUSTER=bigtable-test
export BIGTABLE_ZONE=us-central1-c
export SPANNER_INSTANCE=
export COMPOSER_LOCATION=us-central1
export COMPOSER_ENVIRONMENT=
export CLOUD_KMS_KEY=

export MYSQL_INSTANCE=
export MYSQL_USER=
export MYSQL_PASSWORD=
export MYSQL_DATABASE=
export POSTGRES_INSTANCE=
export POSTGRES_USER=
export POSTGRES_PASSWORD=
export POSTGRES_DATABASE=
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We haven't standardized these names across languages/repos, and these names look fine.


# Environment variables for App Engine Flexible system tests.
export GA_TRACKING_ID=
Expand Down
0