Core Python Programming

NOTE: The code covered in this and the previous post are also available in a video walkthrough. Mar 2018 UPDATE: Modernized the code a bit, shortening it, and changed to R/W scope because drive.file doesn't work if the file hasn't been created yet. The same fixes were made to the Drive API v2 sample in the preceding blog post.

Introduction

In a blog post last week, we introduced readers to performing uploads and downloads files to/from Google Drive from a simple Python command-line script. In an official Google blog post later that same day, the Google Drive API team announced a new version of the API. Great timing huh? Well, good thing I knew it was coming, so that I could prepare this post for you, which is a primer on how to migrate from the current version of the API (v2) to the new one (v3).

As stated by the Drive team, v2 isn't being deprecated, and there are no new features in v3, thus migration isn't required. The new version is mainly for new apps/integrations as well as developers with v2 apps who wish to take advantage of the improvements. This post is intended for those in the latter group, covering porting existing apps to v3. Ready? Let's go straight to the action.

Migrating from Google Drive API v2 to v3

Most of this post will be just examining all the "diffs" between the v2 code sample from the previous post (renamed from drive_updown.py to drive_updown2.py) and its v3 equivalent (drive_updown3.py). We'll take things step-by-step to provide more details, but let's start with all the diffs first:

--- drive_updown2.py   2018-03-11 21:42:33.000000000 -0700
+++ drive_updown3.py   2018-03-11 21:44:57.000000000 -0700
@@ -11,23 +11,24 @@
 if not creds or creds.invalid:
     flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
     creds = tools.run_flow(flow, store)
-DRIVE = discovery.build('drive', 'v2', http=creds.authorize(Http()))
+DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))
 
 FILES = (
-    ('hello.txt', False),
-    ('hello.txt', True),
+    ('hello.txt', None),
+    ('hello.txt', 'application/vnd.google-apps.document'),
 )
 
-for filename, convert in FILES:
-    metadata = {'title': filename}
-    res = DRIVE.files().insert(convert=convert, body=metadata,
-            media_body=filename, fields='mimeType,exportLinks').execute()
+for filename, mimeType in FILES:
+    metadata = {'name': filename}
+    if mimeType:
+        metadata['mimeType'] = mimeType
+    res = DRIVE.files().create(body=metadata, media_body=filename).execute()
     if res:
         print('Uploaded "%s" (%s)' % (filename, res['mimeType']))
 
 if res:
     MIMETYPE = 'application/pdf'
-    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
+    data = DRIVE.files().export(fileId=res['id'], mimeType=MIMETYPE).execute()
     if data:
         fn = '%s.pdf' % os.path.splitext(filename)[0]
         with open(fn, 'wb') as fh:

We'll start with the building of the service endpoint, with the trivial change of the API version string from 'v2' to 'v3':

-DRIVE = build('drive', 'v2', http=creds.authorize(Http()))
+DRIVE = build('drive', 'v3', http=creds.authorize(Http()))

The next change is the deprecation of the conversion flag. The problem with a Boolean variable is that it limits the possible types of file formats supported. By changing it to a file mimeType instead, the horizons are broadened:

 FILES = (
-    ('hello.txt', False),
-    ('hello.txt', True),
+    ('hello.txt', None),
+    ('hello.txt', 'application/vnd.google-apps.document'),
 )

Your next question will be: "What are the mimeTypes for the supported Google Apps document formats?" The answers can be found at this page in the official docs. This changes the datatype in our array of 2-tuples, so we need to change the loop variable to reflect this... we'll use the mimeType instead of a conversion flag:

-for filename, convert in FILES:
+for filename, mimeType in FILES:

Another change related to deprecating the convert flag is that the mimeType isn't a parameter to the API call. Instead, it's another piece of metadata, so we need to add mimeType to the metadata object.

Related to this is a name change: since a file's name is its name and not its title, it makes more sense to use "name" as the metadata value:

-    metadata = {'title': filename}
+    metadata = {'name': filename}
+    if mimeType:
+        metadata['mimeType'] = mimeType

Why the if statement? Not only did v3 see a change to using mimeTypes, but rather than being a parameter like the conversion flag in v2, the mimeType has been moved into the file's metadata, so if we're doing any conversion, we need to add it to our metadata field (then remove the convert parameter down below).

Next is yet another name change: when creating files on Google Drive, "create()" makes more sense as a method name than "insert()". Reducing the size of payload is another key ingredient of v3. We mentioned in the previous post that insert() returns more than 30 fields in the response payload unless you use the fields parameter to specify exactly which you wish returned. In v3, the default response payload only returns four fields, including all the ones we need in this script, so use of the fields parameter isn't required any more:

-    res = DRIVE.files().insert(convert=convert, body=metadata,
-            media_body=filename, fields='mimeType,exportLinks').execute()
+    res = DRIVE.files().create(body=metadata, media_body=filename).execute()

The final improvement we can demonstrate: users no longer have to make an authorized HTTP GET request with a link to export and download a file in an alternate format like PDF®. Instead, it's now a "normal" API call (to the new "export()" method) with the mimeType as a parameter. The only other parameter you need is the file ID, which comes back as part of the (default) response payload when the create() call was made:

-    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
+    data = DRIVE.files().export(fileId=res['id'], mimeType=MIMETYPE).execute()

That's it! If you run the script, grant the script access to your Google Drive (via the OAuth2 prompt that pops up in the browser), and then you should get output that looks like this:

$ python drive_updown3.py # or python3
Uploaded "hello.txt" (text/plain)
Uploaded "hello.txt" (application/vnd.google-apps.document)
Downloaded "hello.pdf" (application/pdf)

Conclusion

The entire v2 script (drive_updown2.py) was spelled out in full in the previous post, and it hasn't changed since then. Below is the v3 script (drive_updown3.py) for your convenience which runs on both Python 2 and Python 3 (unmodified!):

#!/usr/bin/env python

from __future__ import print_function
import os

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

FILES = (
    ('hello.txt', None),
    ('hello.txt', 'application/vnd.google-apps.document'),
)

for filename, mimeType in FILES:
    metadata = {'name': filename}
    if mimeType:
        metadata['mimeType'] = mimeType
    res = DRIVE.files().create(body=metadata, media_body=filename).execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

if res:
    MIMETYPE = 'application/pdf'
    data = DRIVE.files().export(fileId=res['id'], mimeType=MIMETYPE).execute()
    if data:
        fn = '%s.pdf' % os.path.splitext(filename)[0]
        with open(fn, 'wb') as fh:
            fh.write(data)
        print('Downloaded "%s" (%s)' % (fn, MIMETYPE))

)

Just as in the previous post(s), you can now customize this code for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs. Hope we accomplished our goal by pointing out some of the shortcomings that are in v2 and how they were improved in v3! All of the content in this and the previous post are spelled out visually in this video that I created for you.

UPDATE: Since this post was published, the Google Drive team released a newer version of their API. After reading this one, go to the next post to learn about migrating your app from v2 to v3 as well as link to my video which walks through the code samples in both posts.

Introduction

So far in this series of blogposts covering authorized Google APIs, we've used Python to access Google Drive, Gmail, and Google Calendar. Today, we're revisiting Google Drive with a small snippet that uploads plain text files to Drive, with & without conversion to a Google Apps format (Google Docs), then exports & downloads the converted one as PDF®.

Earlier posts demonstrated the structure and "how-to" use Google APIs in general, so more recent posts, including this one, focus on solutions and apps, and use of specific APIs. Once you review the earlier material, you're ready to start with authorization scopes then see how to use the API itself.

Google Drive API Scopes

Google Drive features numerous API scopes of authorization. As usual, we always recommend you use the most restrictive scope possible that allows your app to do its work. You'll request fewer permissions from your users (which makes them happier), and it also makes your app more secure, possibly preventing modifying, destroying, or corrupting data, or perhaps inadvertently going over quotas. Since we need to upload/create files in Google Drive, the minimum scope we need is:

• 'https://www.googleapis.com/auth/drive' — Read/write access to Drive

Using the Google Drive API

Let's get going with our example today that uploads and downloads a simple plain text file to Drive. The file will be uploaded twice, once as-is, and the second time, converted to a Google Docs document. The last part of the script will request an export of the (uploaded) Google Doc as PDF and download that from Drive.

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, we're going to skip that here and jump right to the action, creating of a service endpoint to Drive. The API name is (of course) 'drive', and the current version of the API is 2, so use the string 'v2' in this call to the apiclient.discovey.build() function:

DRIVE = build('drive', 'v2', http=creds.authorize(Http()))

Let's also create a FILES array object (tuple, list, etc.) which holds 2-tuples of the files to upload. These pairs are made up of a filename and a flag indicating whether or not you wish the file to be converted to a Google Apps format:

FILES = (
    ('hello.txt', False),
    ('hello.txt', True),
)

Since we're uploading a plain text file, a conversion to Apps format means Google Docs. (You can imagine that if it was a CSV file, the target format would be Google Sheets instead.) With the setup complete, let's move on to the code that performs the file uploads.

We'll loop through FILES, cycling through each file-convert flag pair and call the files.insert() method to perform the upload. The four parameters needed are: 1) the conversion flag, 2) the file metadata, which is only the filename (see below), 3) the media_body, which is also the filename but has a different purpose — it specifies where the file content will come from, meaning the file will be opened and its data transferred to the API, and 4), a set of fields you want returned.

for filename, convert in FILES:
    metadata = {'title': filename}
    res = DRIVE.files().insert(convert=convert, body=metadata,
            media_body=filename, fields='mimeType,exportLinks').execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

It's important to give the fields() parameter because if you don't, more than 30(!) are returned by default from the API. There's no need to waste all that network traffic if all you need are just a couple. In our case, we only want the mimeType, to confirm what the file was saved as, and exportLinks, which we'll explore in a moment. If files are uploaded successfully, the print() lets the user know, and then we move on to the final section of the script.
Before we dig into the last bit of code, it's important to realize that the res variable still contains the result from the second upload, the one where the file is converted to Google Docs. This is important because this is where we need to extract the download link for the format you want (res['exportLinks'][MIMETYPE]). The way to download the file is to make an authorized HTTP GET call, passing in that link. In our case, it's the PDF version. If the download is successful, the data variable will have the payload to write to disk. If all's good, let the user know:

if res:
    MIMETYPE = 'application/pdf'
    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
    if data:
        fn = '%s.pdf' % os.path.splitext(filename)[0]
        with open(fn, 'wb') as fh:
            fh.write(data)
        print('Downloaded "%s" (%s)' % (fn, MIMETYPE))

Final note: this code sample is slightly different from previous posts in two big ways: 1) now that the Google APIs Client Library runs on Python 3, I'll try to produce only code samples for this blog that run unmodified under both 2.x and 3.x interpreters — the primary one-line difference being the import of the print() function, and 2) we're going to incorporate the use of the run_flow() function from oauth2client.tools and only fallback to the deprecated run() function if necessary — more info on this change available in this earlier post.

If you run the script, grant the script access to your Google Drive (via the OAuth2 prompt that pops up in the browser), and then you should get output that looks like this:

$ python drive_updown3.py # or python3
Uploaded "hello.txt" (text/plain)
Uploaded "hello.txt" (application/vnd.google-apps.document)
Downloaded "hello.pdf" (application/pdf)

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

#!/usr/bin/env python

from __future__ import print_function
import os

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v2', http=creds.authorize(Http()))

FILES = (
    ('hello.txt', False),
    ('hello.txt', True),
)

for filename, convert in FILES:
    metadata = {'title': filename}
    res = DRIVE.files().insert(convert=convert, body=metadata,
            media_body=filename, fields='mimeType,exportLinks').execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

if res:
    MIMETYPE = 'application/pdf'
    res, data = DRIVE._http.request(res['exportLinks'][MIMETYPE])
    if data:
        fn = '%s.pdf' % os.path.splitext(filename)[0]
        with open(fn, 'wb') as fh:
            fh.write(data)
        print('Downloaded "%s" (%s)' % (fn, MIMETYPE))

You can now customize this code for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs. If you want to see another example of using the Drive API, check out this earlier post listing the files in Google Drive and its accompanying video as well as a similar example in the official docs or its equivalent in Java (server-side, Android), iOS (Objective-C, Swift), C#/.NET, PHP, Ruby, JavaScript (client-side, Node.js, Google Apps Script), or Go. That's it... hope you find these code samples useful in helping you get started with the Drive API!

UPDATE: Since this post was published, the Google Drive team released a newer version of their API. Go to the next post to learn about migrating your app from v2 to v3 as well as link to my video which walks through the code samples in both posts.

EXTRA CREDIT: Feel free to experiment and try something else to test your skills and challenge yourself as there's a lot more to Drive than just uploading and downloading files. Experiment with creating folders and manipulate files there, work with a folder of photos and organize them using the image metadata available to you, implement a search engine for your Drive files, etc. There are so many things you can do! 

In this post, we're going to give you a user-friendly introduction to Python decorators. (The code works on both Python 2 [2.6 or 2.7 only] and 3 so don't be concerned with your version.) Before jumping into the topic du jour, consider the usefulness of the map() function. You've got a list with some data and want to apply some function [like times2() below] to all its elements and get a new list with the modified data:

def times2(x):
    return x * 2

>>> list(map(times2, [0, 1, 2, 3, 4]))
[0, 2, 4, 6, 8]

Yeah yeah, I know that you can do the same thing with a list comprehension or generator expression, but my point was about an independent piece of logic [like times2()] and mapping that function across a data set ([0, 1, 2, 3, 4]) to generate a new data set ([0, 2, 4, 6, 8]). However, since mapping functions like times2()aren't tied to any particular chunk of data, you can reuse them elsewhere with other unrelated (or related) data.

Along similar lines, consider function calls. You have independent functions and methods in classes. Now, think about "mapped" execution across functions. What are things that you can do with functions that don't have much to do with the behavior of the functions themselves? How about logging function calls, timing them, or some other introspective, cross-cutting behavior. Sure you can implement that behavior in each of the functions that you care about such information, however since they're so generic, it would be nice to only write that logging code just once.

Two years ago, I wrote a post on "learning Python" to launch this blog dedicated to Python. While useful, it doesn't address beginners' needs as much, so it's time for a revisit. Because Python is such a user-friendly language for beginners, I'm often asked whether Python is the "best first language" for those new to programming. While tempted to respond in the affirmative, my answer really is, "it depends." It depends on your experience, age, level of exposure, etc.

Yes, there are indeed plenty of resources out there, such as courses from online learning brands such as Khan Academy, Udacity, Coursera, Codecademy, CodeSchool, and edX, but most certainly don't come with an instructor, instead relying on live or recorded videos and possibly supplemental study groups, or "cohort learning," as a colleague of mine has branded it. Whatever the mechanism, it's surely better than pure online tutorials or slaving away over a book, neither of which come with instructors either.

Stepping back a bit, before jumping into hardcore C/C++, Java, PHP, Ruby, or Javascript lessons, for learning tools that are used in industry today, there are better stepping stones to get you there. You may be a kid or a professional who either doesn't code much or had done so long ago. You're say that type of user who is "insulted" by the move "left" or "right" commands for controlling a turtle, say, and desire something more complex. The good news is that there are tools out there, more which allow you to venture further without an instructor.

One of them is Scratch, a "jigsaw puzzle"-like programming language created at MIT (or Tynker or Blockly, Scratch-like derivatives). Yes, you will do left, right, up, down, etc., but you'll also get to play audio, video, repeat commands, draw graphics, and make sounds. This tool is great for teaching the young learner, who don't need any of the advanced features but which are available for when they're ready to take the next step. It can be used to teach children the concepts of programming without all the syntax that text-based programming languages feature which may make learning those concepts a burden.

If you wish to proceed, go to the website to get started. They've got videos there as well as projects you can copy. As you can see, you snap together puzzle pieces that teach you coding. Better yet, to get started even more quickly, clone one (or more) of the projects, and "tweak" the code a bit to "do your own thing." In time, you may even develop your own fun applications or real games. Another similar graphical learning tool to consider is Alice from the University of Virginia and now Carnegie-Mellon University.

Once you're comfortable with that type of working environment, there's a similar tool from MIT called App Inventor. Leverage your Scratch skills and start building applications that run on Android devices! There's an emulator, so you don't really need an Android device, but it's certainly more rewarding when you can use an app that you built running on a tablet or phone! (Try a family friend who may have an old device they don't use any more.)

Once you're to move beyond block-like languages, there are 2 good choices (or better yet, do both!). One of which the de facto language of the web: Javascript. Unfortunately, there are so many online tutorials out there, I wouldn't know which to suggest, so looking forward to your comments below. The ones which are the most effective however, have you learning then coding directly into the browser and seeing results immediately, requiring you to write successful Javascript before allowing you to move on.

The thing about Javascript is that code typically only runs within the browser, to control web pages (i.e., "DOM manipulation") and actions you can take on a single page -- it can also be AJAX code that makes an external call to update a page without requiring a page load. Nevertheless, browser-only execution can be somewhat limiting, so there are now 2 additional ways you can use it.

One is to write "server-side" applications via Node.js. This type of Javascript allows you to write code that executes on the remote machine serving your web pages (generally) after you've entered information in a form and clicked submit. For every web page that users see and interact with, there's also got to be code on the server side that does all the work! This code will also end up returning the final HTML that users see in their browsers once the form has been submitted and results returned.

Another place you can use Javascript is in Google's cloud. The tool there is called Google Apps Script. Using Apps Script, you can create applications that interact with various Google Apps, automate repetitive tasks, or write glue code that lets you connect and share data between different Google services. Try some of their tutorials to get started!

The other option besides Javascript is Python. No doubt you already know what it is since you're here. Python's syntax is extremely approachable for beginners and is widely considered "executable pseudocode." That's right, a programming language that doesn't require you have a Computer Science degree to make good use of it! It's also one of those rare languages that can be used by adults in the professional world as well as by kids learning how to code. Sure there are many online learning systems out there, a sampling of which are here:

• Codecademy
• LearnPython.org
• PySchools
• Online Python Tutor
• LearnStreet (login optional)
• CodingBat (login optional)
• SingPath (login required)

See if you like any of them or have your new coder friends try them out. However, I think kids (and even adults) learn programming best when they get to write cool games (leveraging the amazing PyGame library). There are several books written just for kids, including "Hello World" which was actually written by an engineer and his son! Along with that book there are two more you should consider:

• Hello World! Computer Programming for Kids and Other Beginners (Sande & Sande)
• Invent your Own Computer Games with Python (Sweigart)
• Python for Kids: A Playful Introduction to Programming (Briggs)

Two of the three books above are in the beginners list I created over a year ago along with two other Python reading lists in this post. (The third book should be added to the list as well.) Those of you who are already programmers probably know which one I would recommend. :-) Seriously though, those reading lists show that I can toot other horns too. :P

Here are other online projects and learning resources, including book websites, that you can also try (many are for kids):

• How to Think Like a Computer Scientist (Downey, Elkner, Meyers, Wentworth) 
• Learning to Program (Gauld)
• LiveWires Python course
• A Byte of Python (Swaroop)
• Instant Hacking: Learning to Program with Python (Hetland)
• Snake Wrangling for Kids (Briggs)
• Computer Programming is Fun! (Handy)
• Guido van Robot
• RUR-PLE

In conjunction with a good learning system, book, or project-based learning above, you should also try out one of many free online courses to validate things you've picked up but to also build other knowledge you haven't learned yet. There are a pair from Coursera and one from Udacity:

• An Introduction to Interactive Programming in Python (Rice University)
• Learn to Program: The Fundamentals (University of Toronto)
• Introduction to Computer Science (University of Virginia)

For existing programmers who are still questioning why Python, check out Udacity's motivational blogpost.

That's it! Hopefully I've given you enough resources you can pass along to friends and family members who are intrigued by your passion for computer programming and wish to see what all the excitement is all about. A young man I met on vacation this summer motivated this post... good luck Mitchell! I hope to see the rest of you on the road as well, perhaps at a developers' conference or sitting in one of my upcoming Python courses!