Introduction
Are you a developer but a complete beginner using Google APIs? This series is for you because I'm showing you how get started from scratch, beginning with the Google Workspace ("GWS") APIs like Google Drive and Sheets. The first post covered the base requirements for using any Google APIs: authentication ("authn") and authorization ("authz"). The authn requirements are met by having a Google account while the authz requirements are met by having a developer project with appropriate application credentials.
The second post outlined the next steps to using Google APIs: choosing a development language and APIs to use, installing the appropriate client libraries, and enabling the selected API(s) in the DevConsole. For programming language, I picked Python and Node.js and chose the Google Drive API to craft a simple script that dumps out the first 100 files/folders in your Google Drive. Once the Drive API is enabled and client libraries installed, you'll be ready to code.
I'm excited about this third and longest post, not just because it completes the 3-part series, but because I can finally show some code! While I'm most comfortable with Python, I'm also learning Node, so I'm happy to be able to demo the "same" script in both languages. But, let's start with Python first.
Python
This next section describes the major sections of the Python script. The code is described in more detail in the codelab at http://g.co/codelabs/gsuite-apis-intro, however, it's based on the older Python auth libraries whereas all the Drive API documentation has switched to the newer libraries. Regardless, this post give you enough to understand how the script python/drive_list-new.py works.
Imports
from __future__ import print_function
import os.path
from google.auth.transport.requests import Request
from google.oauth2 import credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient import discovery
The standard library imports come first, with __future__.print_function enabling Python 2-3 compatibility by bringing in the Python 3 print() function to Python 2 apps. (This import is ignored in Python 3.) The other brings in file path utilities for managing OAuth tokens.
The next block imports all the packages necessary to talk to GWS APIs, including the Google APIs client library (googleapiclient) and the required security libraries.
Security
Code snippet
creds = None
SCOPES = 'https://www.googleapis.com/auth/drive.metadata.readonly'
TOKENS = 'storage.json'
if os.path.exists(TOKENS):
creds = credentials.Credentials.from_authorized_user_file(TOKENS)
if not (creds and creds.valid):
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'client_secret.json', SCOPES)
creds = flow.run_local_server()
with open(TOKENS, 'w') as token:
token.write(creds.to_json())
Permission scopes
SCOPES represents the permission(s) an app will request from the end-user; it's either a single string or an array of such strings (Python list). In this case, it's one string representing the Google Drive metadata read-only permission scope. While it looks like a URL, it is translated to a sentence in the language specified by your browser's or computer's locale. In English, it would be: "See information about your Google Drive files." For your own apps, the scope(s) will differ depending on what GWS APIs it uses.
For example, I've got another script that accesses the Drive and Sheets APIs as well as the Google Cloud (GCP) Storage and Vision APIs (see this blog post) where the sample app's scopes look like this:
SCOPES = (
'https://www.googleapis.com/auth/drive.readonly',
'https://www.googleapis.com/auth/devstorage.full_control',
'https://www.googleapis.com/auth/cloud-vision',
'https://www.googleapis.com/auth/spreadsheets',
)
In this example, the Drive scope requested differs slightly: yes, it's also read-only, but rather than file metadata, this scope requests permission to access the file content. For GCS and Sheets, both scopes are read-write, and the Cloud Vision API only processes data (doesn't read or write personal data).
OAuth flow
The first post in this series provided the instructions to create an OAuth client ID (and secret), and save those credentials to your local filesystem as client_secret.json. When connecting to Google servers to request API access, they are presented along with the scopes the script is requesting (InstalledAppFlow.from_client_secrets_file()). Once the user permits access, OAuth tokens (creds) are sent back and stored in a local token-storage file (TOKENS). These OAuth tokens are what are required to access Google APIs, and they're cached locally so the user won't be prompted for authz each time the script is executed.
There are a pair of OAuth tokens (access token and refresh token): the access token provides API access but expires while the refresh token doesn't expire (but can be revoked). If the access token has expired, the refresh token is used to request a new (non-expired) access token. More on this entire flow can be found in Google's documentation.
Described just now is how it all works, but this is what you see implemented in the script:
- Check for an existing token storage file, and if so, load its contents
- Check whether credentials (both OAuth2 tokens) exist and are valid
- If credentials exist but expired, use refresh token to request another valid (access) token
- If credentials don't exist, create "OAuth flow" and render to end-user for authz
- Regardless of how credentials were obtained, (re)save latest tokens (last few lines)
Application
DRIVE = discovery.build('drive', 'v3', credentials=creds)
files = DRIVE.files().list().execute().get('files', [])
for f in files: # 4 fields returned: mimeType, kind, id, name
print(f['name'], f['mimeType'])
Funny story: the actual application takes up fewer lines of code than implementing the necessary security described above. The first line creates an endpoint to the Google Drive API (v3 is the current version). Think of this as an "API client." The second line calls list() from the files() collection of methods available via the API and returns the file list. The files are then looped over with filenames and MIMEtypes displayed. If no files/folders are found in the user's Google Drive, an empty array (list) is assigned and nothing is displayed.
That's it, there are no additional lines of code you can't see here. This script along with its older auth library equivalent (drive_list.py) featured in the codelab can be found in the python folder in the repo at https://github.com/wescpy/gsuite-apis-intro. Now let's take a look at the Node versions.
Node.js/JavaScript
The Node.js version of the script is the file, drive_list.js:
Imports
const fs = require('fs').promises;
const path = require('path');
const process = require('process');
const {authenticate} = require('@google-cloud/local-auth');
const {google} = require('googleapis');
The standard Node.js filesystem (fs), filepath (path), and process packages are imported along with the Google auth and API client libraries. If you prefer the ES module style, here are the equivalent import statements you'll find in drive_list.mjs:
import fs from 'node:fs/promises';
import path from 'node:path';
import process from 'node:process';
import {authenticate} from '@google-cloud/local-auth';
import {google} from 'googleapis';
The rest of the code below is identical in both JavaScript versions.
Security
The equivalent JavaScript security code operates exactly like the Python version. The only difference is that the security steps are broken out into their own async functions, so there's a bit more code to look at:
async function loadSavedCredentialsIfExist() {
try {
const content = await fs.readFile(TOKEN_PATH);
const credentials = JSON.parse(content);
return google.auth.fromJSON(credentials);
} catch (err) {
return null;
}
}
async function saveCredentials(client) {
const content = await fs.readFile(CREDENTIALS_PATH);
const keys = JSON.parse(content);
const key = keys.installed || keys.web;
const payload = JSON.stringify({
type: 'authorized_user',
client_id: key.client_id,
client_secret: key.client_secret,
refresh_token: client.credentials.refresh_token,
access_token: client.credentials.access_token,
});
await fs.writeFile(TOKEN_PATH, payload);
}
async function authorize() {
var client = await loadSavedCredentialsIfExist();
if (client) return client;
client = await authenticate({
scopes: SCOPES,
keyfilePath: CREDENTIALS_PATH,
});
if (client.credentials) await saveCredentials(client);
return client;
}
The typical usage pattern is a call to authorize() followed immediately (upon resolution of the Promise) by the code you want to run, in our case, the actual app via the listFiles() function shown below.
Application
async function listFiles(authClient) {
const drive = google.drive({version: 'v3', auth: authClient});
const res = await drive.files.list();
const files = res.data.files || [];
for (let file of files) {
console.log(`${file.name} (${file.mimeType})`);
}
}
authorize().then(listFiles).catch(console.error);
Like the Python version, listFiles() creates a Drive API client and calls its files.list() method. Filenames and MIMEtypes are displayed as long as at least one file/folder is returned. The last line in the script comprises the "main" app, calling authorize() to get the credentials which are passed to listFiles(). That's it, there are no additional lines of code you can't see here. Both scripts can be found in the repo's nodejs folder at https://github.com/wescpy/gsuite-apis-intro.
TypeScript NOTE
Some of you porting this to TypeScript may run into some challenges getting the security code working due to a conflict between
JSONClientandOAuth2Clienttypes. Specifically, this error may show up:error TS2322: Type 'OAuth2Client' is not assignable to type 'JSONClient'. Type 'OAuth2Client' is missing the following properties from type 'Impersonated': sourceClient, targetPrincipal, targetScopes, delegates, and 3 more.The only clue I found touching this topic is in this closed bug on GH. Drop me a comment below if you're able to get a working version, and I'll update this post.
Running the script
Now that coding is complete, it's time to reap the rewards by executing the script. There are two phases of execution. The first is to go through the security process. When complete the code will then run. If no permission changes occur between executions, the security flow should only happen once upon initial execution. After that, the OAuth tokens are "self-managing:"
- If access token still valid, execution proceeds as normal
- If access token expired, refresh token used to request new access token; execution proceeds
Regardless, these steps are not exposed to the end-user who has only to give their initial permissions once. Let's see what these steps look like in practice.
Authn and authz
When executing the script for the first time, users muse authenticate (sign-in to or select a signed-in Google account) then authorize the app, granting it the requested permission(s). For then authn part, you'll get a Google account-chooser from which to select the one you want to use, something that looks like this:
After selecting the desired account, you'll likely get a "Google hasn’t verified this app" screen:
While it may look a bit apprehensive, its purpose is to let users know that Google has not verified this app which will be asking for permissions to access your data. This is all to protect end-users. There's nothing to worry about for now as it's your code and you're only going to access your own data with it. Learn more about this notification in Google's unverified apps support page.
Click on the Advanced link at the bottom which opens up the ability to move on if you understand the risks:
Since you're the developer, hopefully you trust yourself enough to proceed. If so, click on the "Go to YOUR-APP-NAME (unsafe)" link (where YOUR-APP-NAME is what you configured in the OAuth consent screen earlier). That takes you to the long-awaited OAuth2 authz flow dialog:
Here, the end-user is prompted to give the developer the requested permission(s) to access their private data. For the purposes of this demo, you're the end-user, the data owner, and also the developer. Basically, you'll be giving your script access to your Drive files.
When you click Continue, the security flow completes, and your code will execute. If users click Cancel instead, no OAuth tokens are returned, and the script fails; this is all by design. The bottom-line is that because this is private user data, owner permission must be granted before an API access can occur.
To review the difference between authn & authz or to learn more about this security flow, review the Part 1 post as well as this security overview page in the GWS documentation.
Script execution
Once the end-user has opted-in, execution of the script begins. The listFiles() method returns the first 100 files/folders in the end-user's Google Drive, so when running the app, expect to see their names and MIMEtypes. Your output will definitely vary, but below is some sample output from my Google Drive when I crafted this script originally for a post on my old blog:
$ python3 drive_list-new.py # or $ node drive_list.js
Google Maps demo application/vnd.google-apps.spreadsheet
Overview of Google APIs - Sep 2014 application/vnd.google-apps.presentation
tiresResearch.xls application/vnd.google-apps.spreadsheet
6451_Core_Python_Schedule.doc application/vnd.google-apps.document
out1.txt application/vnd.google-apps.document
tiresResearch.xls application/vnd.ms-excel
6451_Core_Python_Schedule.doc application/msword
out1.txt text/plain
Maps and Sheets demo application/vnd.google-apps.spreadsheet
ProtoRPC Getting Started Guide application/vnd.google-apps.document
gtaskqueue-1.0.2_public.tar.gz application/x-gzip
Pull Queues application/vnd.google-apps.folder
gtaskqueue-1.0.1_public.tar.gz application/x-gzip
appengine-java-sdk.zip application/zip
taskqueue.py text/x-python-script
Google Apps Security Whitepaper 06/10/2010.pdf application/pdf
It's not pretty, and my sample Drive folder doesn't even have 100 files/folders in it, but it's just a proof-of-concept and not very flashy. The exciting thing is that you see filenames you recognize, and they're showing up on the command-line, far away from your web browser and the Drive user interface.
For those who prefer more visual content, below is a video I made a few years ago covering much of what's in this post and demonstrating the same (Python) app. Some of the code or video screenshots may have been updated, but the core functionality is mostly identical.
Command-line scripts vs. web or mobile apps
Command-line scripts aren't going to be the most widely implemented type of application. It's more likely you want to add GWS API usage to a web or mobile app. The calls will differ, but the OAuth flow will be similar. Google has a specific page in their docs for managing the OAuth flow process for web apps as well as one for mobile apps.
Summary
The goal of this 3-part blog post series is to start with nothing and bring developers to a fully working command-line script using GWS APIs (e.g., Google Drive). APIs for the GWS products use OAuth client ID as the authz mechanism. Now that you've completed this "Hello World" application, it's up to your imagination of what you can do now... expand Drive API usage, connect with other GWS APIs like for Docs, Sheets, Slides, Forms, Calendar, or Gmail.
In upcoming posts, I'll look at other Google API families which use different kind of authz mechanism, namely API keys (Maps) and service accounts (GCP). Stay tuned for those as well as additional posts on authn and API client libraries. If neither Python or Node.js "are your thing," you'll also find the Google Drive API QuickStart in Java, Go, and client-side JavaScript (see left-nav). Finally, if you have ideas on what you'd like to hear about or how I can improve my Node.js (or Python) code, please drop a comment below!
References
- "Listing your files on Google Drive" codelab
- "Listing your files on Google Drive" video
- Codelab and this post's GitHub repo
- Using Google Workspace APIs: Part 1/3 blog post
- Using Google Workspace APIs: Part 2/3 blog post
WESLEY CHUN, MSCS, is a Google Developer Expert (GDE) in Google Cloud (GCP) & Google Workspace (GWS), author of Prentice Hall's bestselling "Core Python" series, co-author of "Python Web Development with Django", and has written for Linux Journal & CNET. He runs CyberWeb specializing in GCP & GWS APIs and serverless platforms, Python & App Engine migrations, and Python training & engineering. Wesley was one of the original Yahoo!Mail engineers and spent 13+ years on various Google product teams, speaking on behalf of their APIs, producing sample apps, codelabs, and videos for serverless migration and GWS developers. He holds degrees in Computer Science, Mathematics, and Music from the University of California, is a Fellow of the Python Software Foundation, and loves to travel to meet developers worldwide at conferences, user group events, and universities. Follow he/him @wescpy & his technical blog. Find this content useful? Contact CyberWeb if you may need help or buy him a coffee (or tea)!
Top comments (0)