Core Python Programming

NOTE: The code featured here is also available as a video + overview post as part of this developers series from Google.

Introduction

Earlier today at Google I/O, the Hangouts Chat team (including yours truly) delivered a featured talk on the bot framework for Hangouts Chat (talk video here [~40 mins]). If you missed it several months ago, Google launched the new Hangouts Chat application for G Suite customers (but not consumer Gmail users at this time). This next-generation collaboration platform has several key features not available in "classic Hangouts," including the ability to create chat rooms, search, and the ability to attach files from Google Drive. However for developers, the biggest new feature is a bot framework and API allowing developers to create bots that interact with users inside "spaces," i.e., chat rooms or direct messages (DMs).

Before getting started with bots, let's review typical API usage where your app must be authorized for data access, i.e., OAuth2. Once permission is granted, your app calls the API, the API services the request and responds back with the desired results, as illustrated here:

Bot architecture is quite different. With bots, recognize that requests come from users in chat rooms or DMs. Users direct messages towards a bot, then Hangouts Chat relays those requests to the bot. The bot performs all the necessary processing (possibly calling other APIs and services), collates the response, and returns it to Chat, which in turn, displays the results to users. Here's the summarized bot workflow:

A key takeaway from this discussion is that normally in your apps, you call APIs. For bots, Hangouts Chat calls you. For this reason, the service is extremely flexible for developers: you can create bots using pretty much any language (not just Python), using your stack of choice, and hosted on any public or private cloud. The only requirement is that as long as Hangouts Chat can HTTP POST to your bot, you're good to go.

Furthermore, if you think there's something missing in the diagram because you don't see OAuth2 nor API calls, you'd also be correct. Look for both in a future post where I focus on asynchronous responses from Hangouts Chat bots.

Event types & bot processing

So what does the payload look like when your bot receives a call from Hangouts Chat? The first thing your bot needs to do is to determine the type of event received. It can then process the request based on that event type, and finally, return an appropriate JSON payload back to Hangouts Chat to render within the space. There are currently four event types in Hangouts Chat:

1. ADDED_TO_SPACE
2. REMOVED_FROM_SPACE
3. MESSAGE
4. CARD_CLICKED

The first pair are for when a bot is added to or removed from a space. In the first case, the bot would likely send a welcome message like, "Thanks for adding me to this room," and perhaps give some instructions on how to communicate with the bot. When a bot is removed from a space, it can no longer communicate with chat room participants, so there's no response message sent in this case; the most likely action here would be to log that the bot was removed from the space.

The 3rd message type is likely the most common scenario, where a bot has been added to a room, and now a human user is sending it a request. The other common type would be the last one. Rather than a message, this event occurs when users click on a UI card element. The bot's job here is to invoke the "callback" associated with the card element clicked. A number of things can happen here: the bot can return a text message (or nothing at all), a UI card can be updated, or a new card can be returned.

All of what we described above is realized in the pseudocode below (whether you choose to implement in Python or any other supported language) and helpful in preparing you to review the official documentation:

def process_event(req, rsp):
    event = json.loads(req['body']) # event received
    if event['type'] == 'REMOVED_FROM_SPACE':
        # no response as bot removed from room
        return
    elif event['type'] == 'ADDED_TO_SPACE':
        # bot added to room; send welcome message
        msg = {'text': 'Thanks for adding me to this room!'}
    elif event['type'] == 'MESSAGE':
        # message received during normal operations
        msg = respond_to_message(event['message']['text'])
    elif event['type'] == 'CARD_CLICKED':
        # result from user-click on card UI
        action = event['action']
        msg = respond_to_click(
          action['actionMethodName'], action['parameters'])
    else:
        return
    rsp.send(json.dumps(msg))

Regardless of implementation language, you still need to make the necessary tweaks to run on the hosting platform you choose. Our example uses Google App Engine (GAE) for hosting; below you'll see the obvious similarities between our actual bot app and this pseudocode.

Polling chat room users for votes

For a basic voting bot, we need a vote counter, buttons for users to upvote or downvote, and perhaps some reset or "new vote" button. To implement it, we need to unpack the inbound JSON object, determine the event type, then process each event type with actions like this:

1. ADDED_TO_SPACE — ignore.. we don't do anything unless asked by users
2. REMOVED_FROM_SPACE — no action (bot removed)
3. MESSAGE — start new vote
4. CARD_CLICKED — process "upvote", "downvote", or "newvote" request

Here's a handler implementation for Python App Engine using its webapp2 micro framework:

class VoteBot(webapp2.RequestHandler):
    def post(self):
        event = json.loads(self.request.body)
        user = event['user']['displayName']
        if event['type'] == 'CARD_CLICKED':
            method = event['action']['actionMethodName']
            if method == 'newvote':
                body = create_message(user)
            else:
                delta = 1 if method == 'upvote' else -1
                vote_count = int(event['action']['parameters'][0]['value']) + delta
                body = create_message(user, vote_count, True)
        elif event['type'] == 'MESSAGE':
            body = create_message(user)
        else: # no response for ADDED_TO_SPACE or REMOVED_FROM_SPACE
            return
        self.response.headers['Content-Type'] = 'application/json'
        self.response.out.write(json.dumps(body)) 

If you've never written an app using webapp2, don't fret, as this example is easily portable to Flask or other web framework. Why didn't I write it with Flask to begin with? Couple of reasons: 1) the webapp2 framework comes native with App Engine... no need to go download/install it, and 2) because it's native, I don't need to deploy any other files other than bot.py and its app.yaml config file whereas with Flask, you're uploading another ~1300 files in addition this pair. (More on this towards the end of this post.)

Compare this real app to the pseudocode... similar, right? Our bot is a bit more complex in that it renders UI cards, so the create_message() function will need to do more than just return a plain text response. Instead, it must generate and return the JSON markup rendering the card in the Hangouts Chat UI:

def create_message(voter, vote_count=0, should_update=False):
    PARAMETERS = [{'key': 'count', 'value': str(vote_count)}]
    return {
        'actionResponse': {
            'type': 'UPDATE_MESSAGE' if should_update else 'NEW_MESSAGE'
        },
        'cards': [{
            'header': {'title': 'Last vote by %s!' % voter},
            'sections': [{
                'widgets': [{
                    'textParagraph': {'text': '%d votes!' % vote_count}
                }, {
                    'buttons': [{
                        'textButton': {
                            'text': '+1',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'upvote',
                                    'parameters': PARAMETERS,
                                }
                            }
                        }
                    }, {
                        'textButton': {
                            'text': '-1',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'downvote',
                                    'parameters': PARAMETERS,
                                }
                            }
                        }
                    }, {
                        'textButton': {
                            'text': 'NEW',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'newvote',
                                }
                            }
                        }
                    }]
                }]
            }]
        }]
    }

Finishing touches

The last thing App Engine needs is an app.yaml configuration file:

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: bot.app

If you've deployed apps to GAE before, you know you need to create a project in the Google Cloud Developers Console. You also need to have a project for bots, however you can use the same project for both your use of the Hangouts Chat bot framework (link only works for G Suite customers) as well as hosting your App Engine-based bot.

Once you've deployed your bot to GAE, go to the Hangouts Chat API configuration tab, and add the HTTP endpoint for your App Engine-based bot to the Conenctions Settings section:

Conclusion

Congrats for creating your first Python bot! Below is the entire bot.py file for your convenience. (At this time, Python App Engine standard only supports Python 2, so if you want to run Python 3 instead, use the Python App Engine flexible environment.) Also don't forget the app.yaml configuration file above.

import json
import webapp2

def create_message(voter, vote_count=0, should_update=False):
    PARAMETERS = [{'key': 'count', 'value': str(vote_count)}]
    return {
        'actionResponse': {
            'type': 'UPDATE_MESSAGE' if should_update else 'NEW_MESSAGE'
        },
        'cards': [{
            'header': {'title': 'Last vote by %s!' % voter},
            'sections': [{
                'widgets': [{
                    'textParagraph': {'text': '%d votes!' % vote_count}
                }, {
                    'buttons': [{
                        'textButton': {
                            'text': '+1',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'upvote',
                                    'parameters': PARAMETERS,
                                }
                            }
                        }
                    }, {
                        'textButton': {
                            'text': '-1',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'downvote',
                                    'parameters': PARAMETERS,
                                }
                            }
                        }
                    }, {
                        'textButton': {
                            'text': 'NEW',
                            'onClick': {
                                'action': {
                                    'actionMethodName': 'newvote',
                                }
                            }
                        }
                    }]
                }]
            }]
        }]
    }

class VoteBot(webapp2.RequestHandler):
    def post(self):
        event = json.loads(self.request.body)
        user = event['user']['displayName']
        if event['type'] == 'CARD_CLICKED':
            method = event['action']['actionMethodName']
            if method == 'newvote':
                body = create_message(user)
            else:
                delta = 1 if method == 'upvote' else -1
                vote_count = int(event['action']['parameters'][0]['value']) + delta
                body = create_message(user, vote_count, True)
        elif event['type'] == 'MESSAGE':
            body = create_message(user)
        else: # no response for ADDED_TO_SPACE or REMOVED_FROM_SPACE
            return

        self.response.headers['Content-Type'] = 'application/json'
        self.response.out.write(json.dumps(body))

app = webapp2.WSGIApplication([
    ('/', VoteBot),
], debug=True)

Yep, the entire bot is made up of just these 2 files. While porting to Flask is fairly straightforward and Flask apps are supported by App Engine, Flask itself doesn't come with App Engine (quickstart here), so you'll need to upload all of Flask (in addition to those 2 files) to host a Flask version of your bot on GAE.

Both files are also available from this app's GitHub repo which you can fork. I'm happy to entertain PRs for any bugs you find or ways we can simplify this code even more. Below are some additional resources for learning about Hangouts Chat bots and GAE:

• Hangouts Chat bots concepts page
• Creating new Hangouts Chat bots guide
• Hangouts Chat guide to interactive cards
• Other Hangouts Chat sample bots
• Official Hangouts Chat developer documentation—everything you need to know and more!
• Python App Engine (standard) documentation
• Google App Engine product information

Python's a great language to implement bots with because not only can you create something quickly, but choose any platform to host it on, Google App Engine, Amazon EC2/Google Compute Engine, or any cloud that runs Python apps.

Epilogue: code challenge

While usable, this vote bot can be significantly improved. Once you get the basic bot working, I recommend the following enhancements as exercises for the reader:

1. Support vote topics: users starting new votes must state topic in bot message; use as card header
2. Add images (like the Node.js version of our vote bot in the docs hosted on Google Cloud Functions)
3. Track users who have voted (you decide on implementation)
4. Don't let the vote count to go below zero
5. Allow downvotes only from users who have at least one upvote
6. Port your working bot from webapp2 to Flask

NOTE: The code featured here is also available as a video + overview post as part of this developers series from Google.

Introduction

In an earlier post, I introduced Python developers to adding events to users' calendars using the Google Calendar API. However, while being able to insert events is "interesting," it's only half the picture. If you want to give your users a more complete experience, modifying those events is a must-have. In this post, you'll learn how to modify existing events, and as a bonus, learn how to implement repeating events too.

In order to modify events, we need the full Calendar API scope:

• 'https://www.googleapis.com/auth/calendar'—read-write access to Calendar

Skipping the OAuth2 boilerplate, once you have valid authorization credentials, create a service endpoint to the Calendar API like this:

GCAL = discovery.build('calendar', 'v3',
    http=creds.authorize(Http()))

Now you can send the API requests using this endpoint.

Using the Google Calendar API

Our sample script requires an existing Google Calendar event, so either create one programmatically with events().insert() & save its ID as we showed you in that earlier post, or use events().list() or events().get() to get the ID of an existing event.

While you can use an offset from GMT/UTC, such as the GMT_OFF variable from the event insert post, today's code sample "upgrades" to a more general IANA timezone solution. For Pacific Time, it's "America/Los_Angeles". The reason for this change is to allow events that survive across Daylight Savings Time shifts. IOW, a dinner at 7PM/1900 stays at 7PM as we cross fall and spring boundaries. This is especially important for events that repeat throughout the year. Yes, we are discussing recurrence in this post too, so it's particularly relevant.

Modifying calendar events

In the other post, the EVENT body constitutes an "event record" containing the information necessary to create a calendar entry—it consists of the event name, start & end times, and invitees. That record is an API resource which you created/accessed with the Calendar API via events().insert(). (What do you think the "R" in "URL" stands for anyway?!?) The Calendar API adheres to RESTful semantics in that the HTTP verbs match the actions you perform against a resource.

In today's scenario, let's assume that dinner from the other post didn't work out, but that you want to reschedule it. Furthermore, not only do you want to make that dinner happen again, but because you're good friends, you've made a commitment to do dinner every other month for the rest of the year, then see where things stand. Now that we know what we want, we have a choice.

There are two ways to modifying existing events in Calendar:

1. events().patch() (HTTP PATCH)—"patch" 1 or more fields in resource
2. events().update() (HTTP PUT)—replace/rewrite entire resource

Do you just update that resource with events().patch() or do you replace the entire resource with events().update()? To answer that question, ask yourself, "How many fields am I updating?" In our case, we only want to change the date and make this event repeat, so PATCH is a better solution. If instead, you also wanted to rename the event or switch dinner to another set of friends, you'd then be changing all the fields, so PUT would be a better solution in that case.

Using PATCH means you're just providing the deltas between the original & updated event, so the EVENT body this time reflects just those changes:

TIMEZONE = 'America/Los_Angeles'
EVENT = {
    'start':  {'dateTime': '2017-07-01T19:00:00', 'timeZone': TIMEZONE},
    'end':    {'dateTime': '2017-07-01T22:00:00', 'timeZone': TIMEZONE},
    'recurrence': ['RRULE:FREQ=MONTHLY;INTERVAL=2;UNTIL=20171231']
}

Repeating events

Something you haven't seen before is how to do repeating events. To do this, you need to define what’s known as a recurrence rule ("RRULE"), which answers the question of how often an event repeats. It looks somewhat cryptic but follows the RFC 5545 Internet standard which you can basically decode like this:

• FREQ=MONTHLY—event to occur on a monthly basis...
• INTERVAL=2—... but every two months (every other month)
• UNTIL=20171231—... until this date

There are many ways events can repeat, so I suggest you look at all the examples at the RFC link above.

Finishing touches

Finally, provide the EVENT_ID and call events().patch():

EVENT_ID = YOUR_EVENT_ID_STR_HERE # use your own!
e = GCAL.events().patch(calendarId='primary', eventId=EVENT_ID,
        sendNotifications=True, body=EVENT).execute()

Keep in mind that in real life, your users may be accessing your app from their desktop or mobile devices, so you need to ensure you don't override an earlier change. In this regard, developers should use the If-Match header along with an ETag value to validate unique requests. For more information, check out the conditional modification page in the official docs.

The one remaining thing is to confirm on-screen that the calendar event was updated successfully. We do that by checking the return value—it should be an Event object with all the existing details as well as the modified fields:

print('''\
*** %r event (ID: %s) modified:
    Start: %s
    End:   %s
    Recurring (rule): %s
''' % (e['summary'].encode('utf-8'), e['id'], e['start']['dateTime'],
        e['end']['dateTime'], e['recurrence'][0]))

That's pretty much the entire script save for the OAuth2 boilerplate code we've explored previously. The script is posted below in its entirety, and if you add a valid event ID and run it, depending on the date/times you use, you'll see something like this:

$ python gcal_modify.py
*** 'Dinner with friends' event (ID: YOUR_EVENT_ID_STR_HERE) modified:
    Start: 2017-07-01T19:00:00-07:00
    End:   2017-07-01T22:00:00-07:00
    Recurring (rule): RRULE:FREQ=MONTHLY;UNTIL=20171231;INTERVAL=2

It also works with Python 3 with one slight nit/difference being the "b" prefix on from the event name due to converting from Unicode to bytes:
*** b'Dinner with friends' event...

Conclusion

Now you know how to modify events as well as make them repeat. To complete the example, below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

from __future__ import print_function
from apiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/calendar'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
CAL = build('calendar', 'v3', http=creds.authorize(Http()))

TIMEZONE = 'America/Los_Angeles'
EVENT = {
    'start':  {'dateTime': '2017-07-01T19:00:00', 'timeZone': TIMEZONE},
    'end':    {'dateTime': '2017-07-01T22:00:00', 'timeZone': TIMEZONE},
    'recurrence': ['RRULE:FREQ=MONTHLY;INTERVAL=2;UNTIL=20171231']
}
EVENT_ID = YOUR_EVENT_ID_STR_HERE
e = GCAL.events().patch(calendarId='primary', eventId=EVENT_ID,
        sendNotifications=True, body=EVENT).execute()

print('''\
*** %r event (ID: %s) modified:
    Start: %s
    End:   %s
    Recurring (rule): %s
''' % (e['summary'].encode('utf-8'), e['id'], e['start']['dateTime'],
        e['end']['dateTime'], e['recurrence'][0]))

You can now customize this code for your own needs, for a mobile frontend, a server-side backend, or to access other Google APIs. If you want to learn more about using the Google Calendar API, check out the following resources:

• Creating great experiences with email markup—video
• Creating events with the Calendar API—video & blog post
• Creating events with the Calendar API in Python—blog post
• Official Calendar API Python quickstart—list the next 10 events in your calendar
• Official Calendar API documentation—everything you need to know and more!
• Email markup documentation for Google Calendar

2023 UPDATE: We are working to put updated versions of all the code into GitHub... stay tuned. The link will provided in all posts once the code sample(s) is(are) available.

NOTE: The content here is also available as a video and overview post, part of this series.

UPDATE (Feb 2017): Tweaked the code sample as the isPrimary flag may be missing from non-primary aliases; also added link above to video.

Introduction

In a previous post, I introduced Python developers to the Gmail API with a tutorial on how to search for threads with a minimum number of messages. Today, we'll explore another part of the API, covering the settings endpoints that were added in mid-2016. What's the big deal? Well, you couldn't use the API to read nor modify user settings before, and now you can!

One example all of us can relate to is your personal email signature. Wouldn't it be great if we could modify it programmatically, say to include some recent news about you (perhaps a Tweet other social post), or maybe some random witty quote? You could then automate it to change once a quarter, or even hourly if you like being truly random!

Using the Gmail API

Our simple Python script won't be sending email nor reading user messages, so the only authorization scope needed is the one that accesses basic user settings (there's another for more sensitive user settings):

• https://www.googleapis.com/auth/gmail.settings.basic — Manage basic Gmail user settings

See the documentation for a list of all Gmail API scopes and what each of them mean. Since we've fully covered the authorization boilerplate in earlier posts and videos, including how to connect to the Gmail API, we're going to skip that here and jump right to the action. You can copy the boilerplate from other scripts you've written. Regardless, be sure to create an service endpoint to the API:

GMAIL = discovery.build('gmail', 'v1',
    http=creds.authorize(Http()))

What are "sendAs" email addresses?

First, a quick word about "sendAs" email addresses. Gmail lets you send email from addresses other than your actual Gmail address (considered your primary address). This lets you manage multiple accounts from the same Gmail user interface. (As expected, you need to own or otherwise have access to the alternate email addresses in order to do this.) However, most people only use their primary address, so you may not know about it. You can learn more about sendAs addresses here and here.

Now you may be tempted to use the term "alias," especially because that word was mentioned in those Help pages you just looked at right? However for now, I'd recommend trying to avoid that terminology as it refers to something else in a G Suite/Google Apps context. Can't you see how we already got distracted from the core reason for this post? See, you almost forgot about email signatures already, right? If you stick with "sender addresses" or "sendAs email addresses," there won’t be any confusion.

Using a "Quote of the Day" in your email signature

The Python script we're exploring in this post sets a "Quote of the Day" (or "QotD" for short) as the signature of your primary sendAs address. Where does the QotD come from? Well, it can be as simple (and boring) as this function that returns a hardcoded string:



Cute but not very random right? A better idea is to choose from a number of quotes you have in a relational database w/columns for quotes & authors. Here’s some sample code for data in a SQLite database:



More random, which is cool, but this particular snippet isn't efficient because we’re selecting all rows and then choosing a quote randomly. Obviously there's a better way if a database is your data source. I prefer using a web service instead, coming in the form of a REST API. The code snippet here does just that:



You only need to find a quote-of-the-day service and provide its URL on line 8 that returns a JSON payload. Obviously you'll need a bit more scaffolding if this were a real service, but in this pseudocode example, you can assume that using urllib.{,request.}urlopen() works where the service sends back an empty string upon failure. To play it safe, this snippet falls back to the hardcoded string we saw earlier if the service doesn't return a quote, which comes back as a 2-tuple representing quote and author, respectively.

Setting your new email signature

Now that we're clear on the source for the QotD, we can focus on actually setting it as your new email signature. To do that, we need to get all of your sender (sendAs email) addresses—the goal is only to change your primary addresses (and none of the others if you have any):

addresses = GMAIL.users().settings().sendAs().list(userId='me',
    fields='sendAs(isPrimary,sendAsEmail)').execute().get('sendAs')

As in our other Gmail example, a userId of 'me' indicates the currently-authenticated user. The API will return a number of attributes. If know exactly which ones we want, we can specify them in with the fields attribute so as to control size of the return payload which may contribute to overall latency. In our case, we're requesting just the sendAs.isPrimary flag and sendAs.sendAsEmail, the actual email address string of the sender addresses. What's returned is a Python list consisting of all of your sendAs email addresses, which we cycle through to find the primary address:

for address in addresses:
    if address.get('isPrimary'):
        break

One of your sender addresses must be primary, so unless there's a bug in Gmail, when control of the for loop concludes, address will point to your primary sender address. Now all you have to do is set the signature and confirm to the user:

rsp = GMAIL.users().settings().sendAs().patch(userId='me',
        sendAsEmail=address['sendAsEmail'], body=DATA).execute()
print("Signature changed to '%s'" % rsp['signature'])

If you only have one sender address, there's no need request all the addresses and loop through them looking for the primary address as we did above. In such circumstances, that entire request and loop are extraneous... just pass your email address as the sendAsEmail argument, like this:

rsp = GMAIL.users().settings().sendAs().patch(userId='me',
        sendAsEmail=YOUR_EMAIL_ADDR_HERE, body=DATA).execute()

Conclusion

That's all there is... just 26 lines of code. If we use the static string qotd() function above, your output when running this script will look like this:

$ python gmail_change_sig.py # or python3
Signature changed to '"I heart cats."  ~anonymous'
$

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!). By using, copying, and/or modifying this code or any other piece of source from this blog, you implicitly agree to its Apache2 license:

from __future__ import print_function

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

import qotd
DATA = {'signature': qotd.qotd()}   # quote source up-to-you!

SCOPES = 'https://www.googleapis.com/auth/gmail.settings.basic'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
GMAIL = discovery.build('gmail', 'v1', http=creds.authorize(Http()))
# this entire block optional if you only have one sender address
addresses = GMAIL.users().settings().sendAs().list(userId='me',
        fields='sendAs(isPrimary,sendAsEmail)').execute().get('sendAs')
for address in addresses:
    if address.get('isPrimary'):
        break
rsp = GMAIL.users().settings().sendAs().patch(userId='me',
        sendAsEmail=address['sendAsEmail'], body=DATA).execute()
print("Signature changed to '%s'" % rsp['signature'])

As with our other code samples, you can now customize for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs.

Code challenge

Want to exercise your newfound knowledge of using the Gmail API's settings endpoints? Write a script that uses the API to manage filters or configure a vacation responder. HINT: take a look at the official Gmail API docs, including the pages specific to filters and vacation settings.

Introduction

Today, we'll follow-up to my earlier post on the Google Sheets API and multiple posts (first, second, third) on the Google Drive API by answering one common question: How do you download a Google Sheets spreadsheet as a CSV file? The "FAQ"ness of the question itself as well as various versions of Google APIs has led to many similar StackOverflow questions: one, two, three, four, five, just to list a few. Let's answer this question definitively and walk through a Python code sample that does exactly that. The main assumption is that you have a Google Sheet file in your Google Drive named "inventory".

Choosing the right API

Upon first glance, developers may think the Google Sheets API is the one to use. Unfortunately that isn't the case. The Sheets API is the one to use for spreadsheet-oriented operations, such as inserting data, reading spreadsheet rows, managing individual tab/sheets within a spreadsheet, cell formatting, creating charts, adding pivot tables, etc., It isn't meant to perform file-based requests like exporting a Sheet in CSV (comma-separated values) format. For file-oriented operations with a Google Sheet, you would use the Google Drive API.

Using the Google Drive API

As mentioned earlier, Google Drive features numerous API scopes of authorization. As usual, we always recommend you use the most restrictive scope possible that allows your app to do its work. You'll request fewer permissions from your users (which makes them happier), and it also makes your app more secure, possibly preventing modifying, destroying, or corrupting data, or perhaps inadvertently going over quotas. Since we're only exporting a Google Sheets file from Google Drive, the only scope we need is:

• 'https://www.googleapis.com/auth/drive.readonly' — Read-only access to file content or metadata

The earlier post I wrote on the Google Drive API featured sample code that exported an uploaded Google Docs file as PDF and download that from Drive. This post will not only feature a change to exporting a Google Sheets file in CSV format, but also demonstrate one additional feature of the Drive API: querying

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, we're going to skip that here and jump right to the action, creating of a service endpoint to Drive. The API name is (of course 'drive', and the current version of the API is 3, so use the string 'v3' in this call to the apiclient.discovey.build() function:

DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

Query and export files from Google Drive

While unnecessary, we'll create a few string constants representing the filename, source and destination file MIME types to make the code easier to understand:

FILENAME = 'inventory'
SRC_MIMETYPE = 'application/vnd.google-apps.spreadsheet'
DST_MIMETYPE = 'text/csv'

In this simple example, we're only going to export one Google Sheets file as CSV, arbitrarily choosing a file named, "inventory." So to perform the query, you need both the filename and its MIME type, "application/vnd.google-apps.spreadsheet". Query components are conjoined with the "and" keyword, so your query string will look like this: q='name="%s" and mimeType="%s"' % (FILENAME, SRC_MIMETYPE).

Since there may be more than one Google Sheets file named 'inventory". we opt for newest one and thus need to sort all matching files in descending order of last modification time then name if "mtime"s are identical via an "order by" clause: orderBy='modifiedTime desc,name'. Here is the complete call to DRIVE.files().list() to issue the query:

files = DRIVE.files().list(
    q='name="%s" and mimeType="%s"' % (FILENAME, SRC_MIMETYPE),
    orderBy='modifiedTime desc,name').execute().get('files', [])

If any files match, the payload will contain a 'files' key, else we default to an empty list and display to the user on the last line that no files were found. Otherwise, grab the first match, the most recently-modified 'inventory' file, create a suitable CSV filename from it, and change all spaces to underscores:

fn = '%s.csv' % os.path.splitext(files[0]['name'].replace(' ', '_'))[0]

The final Drive API call requests an export of 'inventory' as a CSV file, and if successful, the downloaded data is written with the filename above. In either case, the user is notified of success or failure of the export:

data = DRIVE.files().export(fileId=files[0]['id'], mimeType=DST_MIMETYPE).execute()
if data:
    with open(fn, 'wb') as f:
        f.write(data)
    print('DONE')
else:
    print('ERROR (could not download file)')

Note that if downloading as CSV, the Drive API only exports of the first sheet in a Sheets file... you won't get any others. However, it does support 3 other download formats that will get you all the sheets.

If you create a Sheets file named 'inventory', run the script, grant the script access to your Google Drive (via the OAuth2 prompt that pops up in the browser), and then you should get output that looks like this:

$ python drive_sheets_csv_export.py # or python3
Exporting "inventory" as "inventory.csv"... DONE

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!). If I were to divide the script into 4 major sections, they would be:

• Get creds & build Google Drive service endpoint
• Source and destination file info
• Query Google Drive for matching files
• Export most recent matching Sheets file as CSV

Here's the code itself:

from __future__ import print_function
import os

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive.readonly'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

FILENAME = 'inventory'
SRC_MIMETYPE = 'application/vnd.google-apps.spreadsheet'
DST_MIMETYPE = 'text/csv'

files = DRIVE.files().list(
    q='name="%s" and mimeType="%s"' % (FILENAME, SRC_MIMETYPE),
    orderBy='modifiedTime desc,name').execute().get('files', [])

if files:
    fn = '%s.csv' % os.path.splitext(files[0]['name'].replace(' ', '_'))[0]
    print('Exporting "%s" as "%s"... ' % (files[0]['name'], fn), end='')
    data = DRIVE.files().export(fileId=files[0]['id'], mimeType=DST_MIMETYPE).execute()
    if data:
        with open(fn, 'wb') as f:
            f.write(data)
        print('DONE')
    else:
        print('ERROR (could not download file)')
else:
    print('!!! ERROR: File not found')

As with our other code samples, you can now customize for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs. Hope this helps answer yet another frequently asked question!

NOTE: The code covered in this and the previous post are also available in a video walkthrough. Mar 2018 UPDATE: Modernized the code a bit, shortening it, and changed to R/W scope because drive.file doesn't work if the file hasn't been created yet. The same fixes were made to the Drive API v2 sample in the preceding blog post.

Introduction

In a blog post last week, we introduced readers to performing uploads and downloads files to/from Google Drive from a simple Python command-line script. In an official Google blog post later that same day, the Google Drive API team announced a new version of the API. Great timing huh? Well, good thing I knew it was coming, so that I could prepare this post for you, which is a primer on how to migrate from the current version of the API (v2) to the new one (v3).

As stated by the Drive team, v2 isn't being deprecated, and there are no new features in v3, thus migration isn't required. The new version is mainly for new apps/integrations as well as developers with v2 apps who wish to take advantage of the improvements. This post is intended for those in the latter group, covering porting existing apps to v3. Ready? Let's go straight to the action.

Migrating from Google Drive API v2 to v3

Most of this post will be just examining all the "diffs" between the v2 code sample from the previous post (renamed from drive_updown.py to drive_updown2.py) and its v3 equivalent (drive_updown3.py). We'll take things step-by-step to provide more details, but let's start with all the diffs first:

--- drive_updown2.py   2018-03-11 21:42:33.000000000 -0700
+++ drive_updown3.py   2018-03-11 21:44:57.000000000 -0700
@@ -11,23 +11,24 @@
 if not creds or creds.invalid:
     flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
     creds = tools.run_flow(flow, store)
-DRIVE = discovery.build('drive', 'v2', http=creds.authorize(Http()))
+DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))
 
 FILES = (
-    ('hello.txt', False),
-    ('hello.txt', True),
+    ('hello.txt', None),
+    ('hello.txt', 'application/vnd.google-apps.document'),
 )
 
-for filename, convert in FILES:
-    metadata = {'title': filename}
-    res = DRIVE.files().insert(convert=convert, body=metadata,
-            media_body=filename, fields='mimeType,exportLinks').execute()
+for filename, mimeType in FILES:
+    metadata = {'name': filename}
+    if mimeType:
+        metadata['mimeType'] = mimeType
+    res = DRIVE.files().create(body=metadata, media_body=filename).execute()
     if res:
         print('Uploaded "%s" (%s)' % (filename, res['mimeType']))
 
 if res:
     MIMETYPE = 'application/pdf'
-    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
+    data = DRIVE.files().export(fileId=res['id'], mimeType=MIMETYPE).execute()
     if data:
         fn = '%s.pdf' % os.path.splitext(filename)[0]
         with open(fn, 'wb') as fh:

We'll start with the building of the service endpoint, with the trivial change of the API version string from 'v2' to 'v3':

-DRIVE = build('drive', 'v2', http=creds.authorize(Http()))
+DRIVE = build('drive', 'v3', http=creds.authorize(Http()))

The next change is the deprecation of the conversion flag. The problem with a Boolean variable is that it limits the possible types of file formats supported. By changing it to a file mimeType instead, the horizons are broadened:

 FILES = (
-    ('hello.txt', False),
-    ('hello.txt', True),
+    ('hello.txt', None),
+    ('hello.txt', 'application/vnd.google-apps.document'),
 )

Your next question will be: "What are the mimeTypes for the supported Google Apps document formats?" The answers can be found at this page in the official docs. This changes the datatype in our array of 2-tuples, so we need to change the loop variable to reflect this... we'll use the mimeType instead of a conversion flag:

-for filename, convert in FILES:
+for filename, mimeType in FILES:

Another change related to deprecating the convert flag is that the mimeType isn't a parameter to the API call. Instead, it's another piece of metadata, so we need to add mimeType to the metadata object.

Related to this is a name change: since a file's name is its name and not its title, it makes more sense to use "name" as the metadata value:

-    metadata = {'title': filename}
+    metadata = {'name': filename}
+    if mimeType:
+        metadata['mimeType'] = mimeType

Why the if statement? Not only did v3 see a change to using mimeTypes, but rather than being a parameter like the conversion flag in v2, the mimeType has been moved into the file's metadata, so if we're doing any conversion, we need to add it to our metadata field (then remove the convert parameter down below).

Next is yet another name change: when creating files on Google Drive, "create()" makes more sense as a method name than "insert()". Reducing the size of payload is another key ingredient of v3. We mentioned in the previous post that insert() returns more than 30 fields in the response payload unless you use the fields parameter to specify exactly which you wish returned. In v3, the default response payload only returns four fields, including all the ones we need in this script, so use of the fields parameter isn't required any more:

-    res = DRIVE.files().insert(convert=convert, body=metadata,
-            media_body=filename, fields='mimeType,exportLinks').execute()
+    res = DRIVE.files().create(body=metadata, media_body=filename).execute()

The final improvement we can demonstrate: users no longer have to make an authorized HTTP GET request with a link to export and download a file in an alternate format like PDF®. Instead, it's now a "normal" API call (to the new "export()" method) with the mimeType as a parameter. The only other parameter you need is the file ID, which comes back as part of the (default) response payload when the create() call was made:

-    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
+    data = DRIVE.files().export(fileId=res['id'], mimeType=MIMETYPE).execute()

That's it! If you run the script, grant the script access to your Google Drive (via the OAuth2 prompt that pops up in the browser), and then you should get output that looks like this:

$ python drive_updown3.py # or python3
Uploaded "hello.txt" (text/plain)
Uploaded "hello.txt" (application/vnd.google-apps.document)
Downloaded "hello.pdf" (application/pdf)

Conclusion

The entire v2 script (drive_updown2.py) was spelled out in full in the previous post, and it hasn't changed since then. Below is the v3 script (drive_updown3.py) for your convenience which runs on both Python 2 and Python 3 (unmodified!):

#!/usr/bin/env python

from __future__ import print_function
import os

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

FILES = (
    ('hello.txt', None),
    ('hello.txt', 'application/vnd.google-apps.document'),
)

for filename, mimeType in FILES:
    metadata = {'name': filename}
    if mimeType:
        metadata['mimeType'] = mimeType
    res = DRIVE.files().create(body=metadata, media_body=filename).execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

if res:
    MIMETYPE = 'application/pdf'
    data = DRIVE.files().export(fileId=res['id'], mimeType=MIMETYPE).execute()
    if data:
        fn = '%s.pdf' % os.path.splitext(filename)[0]
        with open(fn, 'wb') as fh:
            fh.write(data)
        print('Downloaded "%s" (%s)' % (fn, MIMETYPE))

)

Just as in the previous post(s), you can now customize this code for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs. Hope we accomplished our goal by pointing out some of the shortcomings that are in v2 and how they were improved in v3! All of the content in this and the previous post are spelled out visually in this video that I created for you.

UPDATE: Since this post was published, the Google Drive team released a newer version of their API. After reading this one, go to the next post to learn about migrating your app from v2 to v3 as well as link to my video which walks through the code samples in both posts.

Introduction

So far in this series of blogposts covering authorized Google APIs, we've used Python to access Google Drive, Gmail, and Google Calendar. Today, we're revisiting Google Drive with a small snippet that uploads plain text files to Drive, with & without conversion to a Google Apps format (Google Docs), then exports & downloads the converted one as PDF®.

Earlier posts demonstrated the structure and "how-to" use Google APIs in general, so more recent posts, including this one, focus on solutions and apps, and use of specific APIs. Once you review the earlier material, you're ready to start with authorization scopes then see how to use the API itself.

Google Drive API Scopes

Google Drive features numerous API scopes of authorization. As usual, we always recommend you use the most restrictive scope possible that allows your app to do its work. You'll request fewer permissions from your users (which makes them happier), and it also makes your app more secure, possibly preventing modifying, destroying, or corrupting data, or perhaps inadvertently going over quotas. Since we need to upload/create files in Google Drive, the minimum scope we need is:

• 'https://www.googleapis.com/auth/drive' — Read/write access to Drive

Using the Google Drive API

Let's get going with our example today that uploads and downloads a simple plain text file to Drive. The file will be uploaded twice, once as-is, and the second time, converted to a Google Docs document. The last part of the script will request an export of the (uploaded) Google Doc as PDF and download that from Drive.

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, we're going to skip that here and jump right to the action, creating of a service endpoint to Drive. The API name is (of course) 'drive', and the current version of the API is 2, so use the string 'v2' in this call to the apiclient.discovey.build() function:

DRIVE = build('drive', 'v2', http=creds.authorize(Http()))

Let's also create a FILES array object (tuple, list, etc.) which holds 2-tuples of the files to upload. These pairs are made up of a filename and a flag indicating whether or not you wish the file to be converted to a Google Apps format:

FILES = (
    ('hello.txt', False),
    ('hello.txt', True),
)

Since we're uploading a plain text file, a conversion to Apps format means Google Docs. (You can imagine that if it was a CSV file, the target format would be Google Sheets instead.) With the setup complete, let's move on to the code that performs the file uploads.

We'll loop through FILES, cycling through each file-convert flag pair and call the files.insert() method to perform the upload. The four parameters needed are: 1) the conversion flag, 2) the file metadata, which is only the filename (see below), 3) the media_body, which is also the filename but has a different purpose — it specifies where the file content will come from, meaning the file will be opened and its data transferred to the API, and 4), a set of fields you want returned.

for filename, convert in FILES:
    metadata = {'title': filename}
    res = DRIVE.files().insert(convert=convert, body=metadata,
            media_body=filename, fields='mimeType,exportLinks').execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

It's important to give the fields() parameter because if you don't, more than 30(!) are returned by default from the API. There's no need to waste all that network traffic if all you need are just a couple. In our case, we only want the mimeType, to confirm what the file was saved as, and exportLinks, which we'll explore in a moment. If files are uploaded successfully, the print() lets the user know, and then we move on to the final section of the script.
Before we dig into the last bit of code, it's important to realize that the res variable still contains the result from the second upload, the one where the file is converted to Google Docs. This is important because this is where we need to extract the download link for the format you want (res['exportLinks'][MIMETYPE]). The way to download the file is to make an authorized HTTP GET call, passing in that link. In our case, it's the PDF version. If the download is successful, the data variable will have the payload to write to disk. If all's good, let the user know:

if res:
    MIMETYPE = 'application/pdf'
    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
    if data:
        fn = '%s.pdf' % os.path.splitext(filename)[0]
        with open(fn, 'wb') as fh:
            fh.write(data)
        print('Downloaded "%s" (%s)' % (fn, MIMETYPE))

Final note: this code sample is slightly different from previous posts in two big ways: 1) now that the Google APIs Client Library runs on Python 3, I'll try to produce only code samples for this blog that run unmodified under both 2.x and 3.x interpreters — the primary one-line difference being the import of the print() function, and 2) we're going to incorporate the use of the run_flow() function from oauth2client.tools and only fallback to the deprecated run() function if necessary — more info on this change available in this earlier post.

If you run the script, grant the script access to your Google Drive (via the OAuth2 prompt that pops up in the browser), and then you should get output that looks like this:

$ python drive_updown3.py # or python3
Uploaded "hello.txt" (text/plain)
Uploaded "hello.txt" (application/vnd.google-apps.document)
Downloaded "hello.pdf" (application/pdf)

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

#!/usr/bin/env python

from __future__ import print_function
import os

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v2', http=creds.authorize(Http()))

FILES = (
    ('hello.txt', False),
    ('hello.txt', True),
)

for filename, convert in FILES:
    metadata = {'title': filename}
    res = DRIVE.files().insert(convert=convert, body=metadata,
            media_body=filename, fields='mimeType,exportLinks').execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

if res:
    MIMETYPE = 'application/pdf'
    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
    if data:
        fn = '%s.pdf' % os.path.splitext(filename)[0]
        with open(fn, 'wb') as fh:
            fh.write(data)
        print('Downloaded "%s" (%s)' % (fn, MIMETYPE))

You can now customize this code for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs. If you want to see another example of using the Drive API, check out this earlier post listing the files in Google Drive and its accompanying video as well as a similar example in the official docs or its equivalent in Java (server-side, Android), iOS (Objective-C, Swift), C#/.NET, PHP, Ruby, JavaScript (client-side, Node.js, Google Apps Script), or Go. That's it... hope you find these code samples useful in helping you get started with the Drive API!

UPDATE: Since this post was published, the Google Drive team released a newer version of their API. Go to the next post to learn about migrating your app from v2 to v3 as well as link to my video which walks through the code samples in both posts.

EXTRA CREDIT: Feel free to experiment and try something else to test your skills and challenge yourself as there's a lot more to Drive than just uploading and downloading files. Experiment with creating folders and manipulate files there, work with a folder of photos and organize them using the image metadata available to you, implement a search engine for your Drive files, etc. There are so many things you can do! 

NOTE: The code covered in this blogpost is also available in a video walkthrough here.

UPDATE (Jan 2016): Tweaked the code to support oauth2client.tools.run_flow() which deprecates oauth2client.tools.run(). You can read more about that change and migration steps here.

Introduction

So far in this series of blogposts covering authorized Google APIs, we've used Python code to access Google Drive and Gmail. Today, we're going to demonstrate the Google Calendar API. While Google Calendar, and calendaring in general, have been around for a long time and are fairly stable, it's somewhat of a mystery as to why so few developers create good calendar integrations, whether it be with Google Calendar, or other systems. We'll try to show it isn't necessarily difficult and hopefully motivate some of you out there to add a calendaring feature in your next mobile or web app.

Earlier posts (link 1, link 2) demonstrated the structure and "how-to" use Google APIs in general, so more recent posts, including this one, focus on solutions and apps, and use of specific APIs. Once you review the earlier material, you're ready to start with authorization scopes then see how to use the API itself.

Google Calendar API Scopes

Below are the Google Calendar API scopes of authorization. There are only a pair (at the time of this writing): read-only and read/write. As usual, use the most restrictive scope you possibly can yet still allowing your app to do its work. This makes your app more secure and may prevent inadvertently going over any quotas, or accessing, destroying, or corrupting data. Also, users are less hesitant to install your app if it asks only for more restricted access to their calendars. However, it's likely that in order to really use the API to its fullest, you will probably have to ask for read-write so that you can add, update, or delete events in their calendars.

• 'https://www.googleapis.com/auth/calendar.readonly' — Read-only access to calendar
• 'https://www.googleapis.com/auth/calendar' — Read/write access to calendar

Using the Google Calendar API

We're going to create a sample Python script that inserts a new event into your Google Calendar. Since this requires modifying your calendar, you need the read/write scope above. The API name is 'calendar' which is currently on version 3, so here's the call to apiclient.discovery.build() you'll use:

GCAL = discovery.build('calendar', 'v3',
    http=creds.authorize(Http()))

Note that all lines of code above that is predominantly boilerplate (that was explained in earlier posts and videos). Anyway, we've got an established service endpoint with build(), we need to come up with the data to create a calendar event with, at the very least, an event name plus start and end times.

Timezone or offset required The API requires either a timezone or a GMT offset, the number of hours your timezone is away from Coordinated Universal Time (UTC, more commonly known as GMT). The format is +/-HH:MM away from UTC. For example, Pacific Daylight Time (PDT, also known as Mountain Standard Time, or MST), is "-07:00," or seven hours behind UTC while Nepal Standard Time (NST [or NPT to avoid confusion with Newfoundland Standard Time]), is "+05:45," or five hours and forty-five minutes ahead of UTC. Also, the offset must be in RFC 3339 format, which implements the specifications of ISO 8601 for the Internet. Timestamps look like the following in the required format: "YYYY-MM-DDTHH:MM:SS±HH:MM". For example, September 15, 2015 at 7 PM PDT is represented by this string: "2015-09-15T19:00:00-07:00". If you wish to avoid offsets and would rather use timezone names instead, see the next post in this series (link at bottom).

The script in this post uses the PDT timezone, so we set the GMT_OFF variable to "-07:00". The EVENT body will hold the event name, and start and end times suffixed with the GMT offset:

GMT_OFF = '-07:00'    # PDT/MST/GMT-7
EVENT = {
    'summary': 'Dinner with friends',
    'start':   {'dateTime': '2015-09-15T19:00:00%s' % GMT_OFF},
    'end':     {'dateTime': '2015-09-15T22:00:00%s' % GMT_OFF},
}

Use the insert() method of the events() service to add the event. As expected, one required parameter is the ID of the calendar to insert the event into. A special value of 'primary' has been set aside for the currently authenticated user. The other required parameter is the event body. In our request, we also ask the Calendar API to send email notifications to the guests, and that's done by passing in the sendNotifications flag with a True value. Our call to the API looks like this:

e = GCAL.events().insert(calendarId='primary',
    sendNotifications=True, body=EVENT).execute()

The one remaining thing is to confirm that the calendar event was created successfully. We do that by checking the return value — it should be an Event object with all the details we passed in a moment ago:

print('''*** %r event added:
    Start: %s
    End:   %s''' % (e['summary'].encode('utf-8'),
        e['start']['dateTime'], e['end']['dateTime']))

Now, if you really want some proof the event was created, one of the fields that's created is a link to the calendar event. We don't use it in the code, but you can... just use e['htmlLink'].

Regardless, that's pretty much the entire script save for the OAuth2 code that we're so familiar with from previous posts. The script is posted below in its entirety, and if you run it, depending on the date/times you use, you'll see something like this:

$ python gcal_insert.py
*** 'Dinner with friends' event added:
    Start: 2015-09-15T19:00:00-07:00
    End:   2015-09-15T22:00:00-07:00

It also works with Python 3 with one slight nit/difference being the "b" prefix on from the event name due to converting from Unicode to bytes:

*** b'Dinner with friends' event added:

Conclusion

There can be much more to adding a calendar event, such as events that repeat with a recurrence rule, the ability to add attachments for an event, such as a party invitation or a PDF of the show tickets. For more on what you can do when creating events, take a look at the docs for events().insert() as well as the corresponding developer guide. All of the docs for the Google Calendar API can be found here. Also be sure to check out the companion video for this code sample. That's it!

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

from __future__ import print_function
from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/calendar'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
GCAL = discovery.build('calendar', 'v3', http=creds.authorize(Http()))

GMT_OFF = '-07:00'      # PDT/MST/GMT-7
EVENT = {
    'summary': 'Dinner with friends',
    'start':  {'dateTime': '2015-09-15T19:00:00%s' % GMT_OFF},
    'end':    {'dateTime': '2015-09-15T22:00:00%s' % GMT_OFF},
    'attendees': [
        {'email': 'friend1@example.com'},
        {'email': 'friend2@example.com'},
    ],
}

e = GCAL.events().insert(calendarId='primary',
        sendNotifications=True, body=EVENT).execute()

print('''*** %r event added:
    Start: %s
    End:   %s''' % (e['summary'].encode('utf-8'),
        e['start']['dateTime'], e['end']['dateTime']))

You can now customize this code for your own needs, for a mobile frontend, a server-side backend, or to access other Google APIs. If you want to see another example of using the Calendar API (listing the next 10 events in your calendar), check out the Python Quickstart example or its equivalent in Java (server-side, Android), iOS (Objective-C, Swift), C#/.NET, PHP, Ruby, JavaScript (client-side, Node.js), or Go. That's it... hope you find these code samples useful in helping you get started with the Calendar API!

Code challenge

To test your skills and challenge yourself, try creating recurring events (such as when you expect to receive your paycheck), events with attachments, or perhaps editing existing events. UPDATE (Jul 2017): If you're ready for the next step, we cover the first and last of those choices in our follow-up post.

NOTE: The code covered in this blogpost is also available in a video walkthrough here.

UPDATE (Aug 2016): The code has been modernized to use oauth2client.tools.run_flow() instead of the deprecated oauth2client.tools.run(). You can read more about that change here.

Introduction

The last several posts have illustrated how to connect to public/simple and authorized Google APIs. Today, we're going to demonstrate accessing the Gmail (another authorized) API. Yes, you read that correctly... "API." In the old days, you access mail services with standard Internet protocols such as IMAP/POP and SMTP. However, while they are standards, they haven't kept up with modern day email usage and developers' needs that go along with it. In comes the Gmail API which provides CRUD access to email threads and drafts along with messages, search queries, management of labels (like folders), and domain administration features that are an extra concern for enterprise developers.

Earlier posts demonstrate the structure and "how-to" use Google APIs in general, so the most recent posts, including this one, focus on solutions and apps, and use of specific APIs. Once you review the earlier material, you're ready to start with Gmail scopes then see how to use the API itself.

Gmail API Scopes

Below are the Gmail API scopes of authorization. We're listing them in most-to-least restrictive order because that's the order you should consider using them in — use the most restrictive scope you possibly can yet still allowing your app to do its work. This makes your app more secure and may prevent inadvertently going over any quotas, or accessing, destroying, or corrupting data. Also, users are less hesitant to install your app if it asks only for more restricted access to their inboxes.

• 'https://www.googleapis.com/auth/gmail.readonly' — Read-only access to all resources + metadata
• 'https://www.googleapis.com/auth/gmail.send' — Send messages only (no inbox read nor modify)
• 'https://www.googleapis.com/auth/gmail.labels' — Create, read, update, and delete labels only
• 'https://www.googleapis.com/auth/gmail.insert' — Insert and import messages only
• 'https://www.googleapis.com/auth/gmail.compose' — Create, read, update, delete, and send email drafts and messages
• 'https://www.googleapis.com/auth/gmail.modify' — All read/write operations except for immediate & permanent deletion of threads & messages
• 'https://mail.google.com/' — All read/write operations (use with caution)

Using the Gmail API

We're going to create a sample Python script that goes through your Gmail threads and looks for those which have more than 2 messages, for example, if you're seeking particularly chatty threads on mailing lists you're subscribed to. Since we're only peeking at inbox content, the only scope we'll request is 'gmail.readonly', the most restrictive scope. The API string is 'gmail' which is currently on version 1, so here's the call to apiclient.discovery.build() you'll use:

GMAIL = discovery.build('gmail', 'v1', http=creds.authorize(Http()))

Note that all lines of code above that is predominantly boilerplate (that was explained in earlier posts). Anyway, once you have an established service endpoint with build(), you can use the list() method of the threads service to request the file data. The one required parameter is the user's Gmail address. A special value of 'me' has been set aside for the currently authenticated user.

threads = GMAIL.users().threads().list(userId='me').execute().get('threads', [])

If all goes well, the (JSON) response payload will (not be empty or missing and) contain a sequence of threads that we can loop over. For each thread, we need to fetch more info, so we issue a second API call for that. Specifically, we care about the number of messages in a thread:

for thread in threads:
    tdata = GMAIL.users().threads().get(userId='me', id=thread['id']).execute()
    nmsgs = len(tdata['messages'])

We're seeking only all threads more than 2 (that means at least 3) messages, discarding the rest. If a thread meets that criteria, scan the first message and cycle through the email headers looking for the "Subject" line to display to users, skipping the remaining headers as soon as we find one:

    if nmsgs > 2:
        msg = tdata['messages'][0]['payload']
        subject = ''
        for header in msg['headers']:
            if header['name'] == 'Subject':
                subject = header['value']
                break
        if subject:
            print('%s (%d msgs)' % (subject, nmsgs))

If you're on many mailing lists, this may give you more messages than desired, so feel free to up the threshold from 2 to 50, 100, or whatever makes sense for you. (In that case, you should use a variable.) Regardless, that's pretty much the entire script save for the OAuth2 code that we're so familiar with from previous posts. The script is posted below in its entirety, and if you run it, you'll see an interesting collection of threads... YMMV depending on what messages are in your inbox:

$ python3 gmail_threads.py
[Tutor] About Python Module to Process Bytes (3 msgs)
Core Python book review update (30 msgs)
[Tutor] scratching my head (16 msgs)
[Tutor] for loop for long numbers (10 msgs)
[Tutor] How to show the listbox from sqlite and make it searchable? (4 msgs)
[Tutor] find pickle and retrieve saved data (3 msgs)

BONUS: Python 3!

As of Mar 2015 (formally in Apr 2015 when the docs were updated), support for Python 3 was added to Google APIs Client Library (3.3+)! This update was a long time coming (relevant GitHub thread), and allows Python 3 developers to write code that accesses Google APIs. If you're already running 3.x, you can use its pip command (pip3) to install the Client Library:

$ pip3 install -U google-api-python-client

Because of this, unlike previous blogposts, we're deliberately going to avoid use of the print statement and switch to the print() function instead. If you're still running Python 2, be sure to add the following import so that the code will also run in your 2.x interpreter:

from __future__ import print_function

Conclusion

To find out more about the input parameters as well as all the fields that are in the response, take a look at the docs for threads().list(). For more information on what other operations you can execute with the Gmail API, take a look at the reference docs and check out the companion video for this code sample. That's it!

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

from __future__ import print_function

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/gmail.readonly'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
GMAIL = discovery.build('gmail', 'v1', http=creds.authorize(Http()))

threads = GMAIL.users().threads().list(userId='me').execute().get('threads', [])
for thread in threads:
    tdata = GMAIL.users().threads().get(userId='me', id=thread['id']).execute()
    nmsgs = len(tdata['messages'])

    if nmsgs > 2:
        msg = tdata['messages'][0]['payload']
        subject = ''
        for header in msg['headers']:
            if header['name'] == 'Subject':
                subject = header['value']
                break
        if subject:
            print('%s (%d msgs)' % (subject, nmsgs))

You can now customize this code for your own needs, for a mobile frontend, a server-side backend, or to access other Google APIs. If you want to see another example of using the Gmail API (displaying all your inbox labels), check out the Python Quickstart example in the official docs or its equivalent in Java (server-side, Android), iOS (Objective-C, Swift), C#/.NET, PHP, Ruby, JavaScript (client-side, Node.js), or Go. That's it... hope you find these code samples useful in helping you get started with the Gmail API!

EXTRA CREDIT: To test your skills and challenge yourself, try writing code that allows users to perform a search across their email, or perhaps creating an email draft, adding attachments, then sending them! Note that to prevent spam, there are strict Program Policies that you must abide with... any abuse could rate limit your account or get it shut down. Check out those rules plus other Gmail terms of use here.