-
Notifications
You must be signed in to change notification settings - Fork 6.6k
Add GCF Python SQL samples #1655
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
722b0fd
Add GCF Python SQL samples
292eaa0
Update test SQL instance name
df9a6ba
Add SQL Kokoro test files
94b76b5
Address comments
f426dda
Switch python samples to pools
8a6e7f8
Use connections for Python MySQL, as libraries don't support pooling
be10b39
Fix lint + tests
686ee00
Fix nits
5530250
Make Python config consistent with Node samples
017b297
Merge branch 'master' into gcf-sql
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# Format: //devtools/kokoro/config/proto/build.proto | ||
|
||
# Download secrets from Cloud Storage. | ||
gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/python-docs-samples" | ||
|
||
# Tell the trampoline which build file to use. | ||
env_vars: { | ||
key: "TRAMPOLINE_BUILD_FILE" | ||
value: "github/python-docs-samples/.kokoro/system_tests.sh" | ||
} | ||
|
||
env_vars: { | ||
key: "NOX_SESSION" | ||
value: "functions and sql and py36 and not venv" | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
# Copyright 2018 Google LLC | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the 'License'); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an 'AS IS' BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
# [START functions_sql_mysql] | ||
from os import getenv | ||
|
||
import pymysql | ||
|
||
is_production = getenv('SUPERVISOR_HOSTNAME') is not None | ||
|
||
# TODO(developer): specify SQL connection details | ||
CONNECTION_NAME = getenv( | ||
'INSTANCE_CONNECTION_NAME', | ||
'<YOUR INSTANCE CONNECTION NAME>') | ||
DB_USER = getenv('MYSQL_USER', '<YOUR DB USER>') | ||
DB_PASSWORD = getenv('MYSQL_PASSWORD', '<YOUR DB PASSWORD>') | ||
DB_NAME = getenv('MYSQL_DATABASE', '<YOUR DB NAME>') | ||
|
||
mysql_config = { | ||
'user': DB_USER, | ||
'password': DB_PASSWORD, | ||
'db': DB_NAME, | ||
'charset': 'utf8mb4', | ||
'cursorclass': pymysql.cursors.DictCursor, | ||
'autocommit': True | ||
} | ||
|
||
if is_production: | ||
mysql_config['unix_socket'] = \ | ||
'/cloudsql/' + CONNECTION_NAME | ||
|
||
# Create SQL connection globally to enable reuse | ||
# PyMySQL does not include support for connection pooling | ||
mysql_conn = pymysql.connect(**mysql_config) | ||
|
||
|
||
def __get_cursor(): | ||
""" | ||
Helper function to get a cursor | ||
PyMySQL does NOT automatically reconnect, | ||
so we must reconnect explicitly using ping() | ||
""" | ||
try: | ||
return mysql_conn.cursor() | ||
except Exception: | ||
mysql_conn.ping(reconnect=True) | ||
return mysql_conn.cursor() | ||
|
||
|
||
def mysql_demo(request): | ||
# Remember to close SQL resources declared while running this function. | ||
# Keep any declared in global scope (e.g. mysql_conn) for later reuse. | ||
with __get_cursor() as cursor: | ||
cursor.execute('SELECT NOW() as now') | ||
results = cursor.fetchone() | ||
return str(results['now']) | ||
# [END functions_sql_mysql] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Copyright 2018 Google LLC | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the 'License'); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an 'AS IS' BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
import mysql_sample | ||
|
||
|
||
def test_mysql(): | ||
mysql_sample.mysql_demo(None) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
# Copyright 2018 Google LLC | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the 'License'); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an 'AS IS' BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
# [START functions_sql_postgres] | ||
from os import getenv | ||
|
||
from psycopg2.pool import SimpleConnectionPool | ||
|
||
is_production = getenv('SUPERVISOR_HOSTNAME') is not None | ||
|
||
# TODO(developer): specify SQL connection details | ||
CONNECTION_NAME = getenv( | ||
'INSTANCE_CONNECTION_NAME', | ||
'<YOUR INSTANCE CONNECTION NAME>') | ||
DB_USER = getenv('POSTGRES_USER', '<YOUR DB USER>') | ||
DB_PASSWORD = getenv('POSTGRES_PASSWORD', '<YOUR DB PASSWORD>') | ||
DB_NAME = getenv('POSTGRES_DATABASE', '<YOUR DB NAME>') | ||
|
||
pg_config = { | ||
'user': DB_USER, | ||
'password': DB_PASSWORD, | ||
'dbname': DB_NAME, | ||
} | ||
|
||
if is_production: | ||
pg_config['host'] = '/cloudsql/' + CONNECTION_NAME | ||
else: | ||
pg_config['host'] = 'localhost' | ||
|
||
# Connection pools reuse connections between invocations, | ||
# and handle dropped or expired connections automatically. | ||
pg_pool = None | ||
|
||
|
||
def postgres_demo(request): | ||
global pg_pool | ||
|
||
# Initialize the pool lazily, in case SQL access isn't needed for this | ||
# GCF instance. Doing so minimizes the number of active SQL connections, | ||
# which helps keep your GCF instances under SQL connection limits. | ||
if not pg_pool: | ||
pg_pool = SimpleConnectionPool(1, 1, **pg_config) | ||
|
||
# Remember to close SQL resources declared while running this function. | ||
# Keep any declared in global scope (e.g. pg_pool) for later reuse. | ||
with pg_pool.getconn().cursor() as cursor: | ||
cursor.execute('SELECT NOW() as now') | ||
results = cursor.fetchone() | ||
return str(results[0]) | ||
# [END functions_sql_postgres] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Copyright 2018 Google LLC | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the 'License'); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an 'AS IS' BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
import postgres_sample | ||
|
||
|
||
def test_postgres(): | ||
postgres_sample.postgres_demo(None) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
psycopg2==2.7.5 | ||
PyMySQL==0.9.2 |
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,13 +1,25 @@ | ||
# Environment variables for system tests. | ||
export GCLOUD_PROJECT=your-project-id | ||
export GOOGLE_CLOUD_PROJECT=$GCLOUD_PROJECT | ||
export FIRESTORE_PROJECT= | ||
|
||
export CLOUD_STORAGE_BUCKET=$GCLOUD_PROJECT | ||
export API_KEY= | ||
export BIGTABLE_CLUSTER=bigtable-test | ||
export BIGTABLE_ZONE=us-central1-c | ||
export SPANNER_INSTANCE= | ||
export COMPOSER_LOCATION=us-central1 | ||
export COMPOSER_ENVIRONMENT= | ||
export CLOUD_KMS_KEY= | ||
ace-n marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
export MYSQL_INSTANCE= | ||
export MYSQL_USER= | ||
export MYSQL_PASSWORD= | ||
export MYSQL_DATABASE= | ||
export POSTGRES_INSTANCE= | ||
export POSTGRES_USER= | ||
export POSTGRES_PASSWORD= | ||
export POSTGRES_DATABASE= | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We haven't standardized these names across languages/repos, and these names look fine. |
||
|
||
# Environment variables for App Engine Flexible system tests. | ||
export GA_TRACKING_ID= | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have any more straightforward methods of detecting the environment? Even better, could we do try/except to simply try to connect to the unix socket and fallback to user settings if it doesn't work? (properly commented)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We ideally should not use
SUPERVISOR_HOSTNAME
. It may not be there in the future. You opt for a separate env var to contain the "key". So..INSTANCE_CONNECTION_TYPE
with eitherunix_socket
(or empty string for default), ofhost
for traditional connections.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Node samples use the
NODE_ENV
env var to determine whether a function has been deployed. For the sake of maintaining patterns across samples, is there a way we can do this here?