DEV Community: Dustin Ingram

Portable Cloud Functions with the Python Functions Framework

Dustin Ingram — Thu, 09 Jan 2020 20:10:46 +0000

In this post, I'll show how the open-source Functions Framework for Python makes Cloud Functions portable across multiple products and ecosystems. I’ll specifically show how to migrate a Python function from Cloud Functions to a service like Cloud Run.

I'll also show you how the Functions Framework gives you the ability to test your functions locally as well.

What are the Functions Frameworks?

The Function Frameworks are open source libraries for writing portable functions -- brought to you by the Google Cloud Functions team.

The Functions Framework lets you write lightweight functions that run in many different environments, including:

Google Cloud Functions
Cloud Run and Cloud Run on GKE
Knative-based environments
Your local development machine
and elsewhere.

You can think of the Functions Framework for a given language as a wrapper around a single function in that language, which handles everything necessary for your function to receive HTTP requests or Cloud Events such as Pub/Sub.

The Functions Framework for Python joins previously open-sourced frameworks for several other popular languages.

Going beyond Cloud Functions

Cloud Functions is a powerful and simple tool for quickly deploying a standalone function that lets you integrate various APIs, products, services, or just respond to events.

However, that simplicity also comes at a cost: the Cloud Functions Runtime makes some opinionated choices about the environment in which your function runs, such as the runtime’s base image, the patch version of Python, and the underlying system libraries that are installed.

For most developers, this is totally fine, as the choices that have been made for the runtime are appropriate for almost all use cases.

However, occasionally you will want to make a small change, such as install a special platform-level package that you need for a particular task.

Or, you'll want to continue using your function as-is, but move it to run on Cloud Run to lower costs.

An example: Portability without the Functions Framework

Let's say we have this minimal Cloud Function:

def hello(request):
    return "Hello world!"

This function doesn't do much: it takes an HTTP request, and returns "Hello world!" as a response.

However, if we wanted to migrate it from Cloud Functions to a service like Cloud Run, which hosts applications, not functions, we'd have to do a lot:

choose a web framework
configure the web application
refactor this function as a view in our web app
add the view as a route

By this point, our "new" function wouldn't look much at all like your original function. If you had chosen Flask as your web framework, it might look something like this:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello world!'

That's a lot of extra boilerplate that you don't need.

An example: Portability with the Functions Framework

With the Functions Framework, none of this refactoring is necessary. You can deploy a function to Cloud Run without changing a single line of code in your main.py file.

Instead, you can define a Dockerfile as follows:

# Use the official Python image.
# https://hub.docker.com/_/python
FROM python:3.7-slim

# Copy local code to the container image.
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . .

# Install production dependencies.
RUN pip install functions-framework
RUN pip install -r requirements.txt

# Run the web service on container startup.
CMD exec functions-framework --target=hello

This Dockerfile largely borrows from the Dockerfile in the Python Cloud Run quickstart, but with two key differences:

it installs the functions-framework dependency
it invokes the Functions Framework in the CMD step.

The last steps are building and deploying the container as you would with any Cloud Run application.

See this entire example on Github here.

A local development experience

Another useful feature of the Functions Framework is the ability to run your Cloud Function locally.

Using the example above, you can run your function on your host machine direction using the Function Framework's handy command-line utility:

$ functions-framework --target hello

The downside to this is that any dependencies need to be installed globally and that there may be conflicts between what your function requires and what your host machine has available.

Ideally, we would isolate our function by building the container image we defined in our Dockerfile and running it locally:

$ docker build -t helloworld . && docker run --rm -p 8080:8080 -e PORT=8080 helloworld

Both of these will start a local development server at http://localhost:8080 that you can send HTTP traffic to.

Bonus: Static Typing

One additional benefit to the Functions Framework is that it makes all of the the necessary types available to statically type your functions.

For example, an HTTP function could be statically typed before:

from typing import Tuple, Union
from flask.wrappers import Request

def my_function(request: Request) -> str:
    ...

But now the Context type is available for background functions:

from typing import Dict
from google.cloud.functions.context import Context

def my_function(data: Dict, context: Context) -> None:
    ...

Next steps

Found this project useful? Give it a star it on Github
Found a bug in the Python Functions Framework? File an issue
Want similar updates? Follow me on Twitter

Using Headless Chrome with Cloud Run

Dustin Ingram — Tue, 22 Oct 2019 18:48:57 +0000

I often see folks trying to use headless Chrome with services like Google Cloud Functions. The phrase "Headless Chrome" might sound very spooky, but it just means the regular Chrome browser, run without a GUI and instead interacted with programatically.

Unfortunately, the necessary Chrome binaries are not installed in the Cloud Functions runtime, and there isn't a way to modify the runtime besides installing Python dependencies.

However, one alternative would be to use Cloud Run, which lets you fully customize the runtime, including installing Chrome! So let's do that.

Configuring: `Dockerfile`

First, we'll create a Dockerfile. This uses the official Python base image, installs some additional dependencies, installs Chrome, and installs the dependencies for our application.



# Use the official Python image.
# https://hub.docker.com/_/python
FROM python:3.7

# Install manually all the missing libraries
RUN apt-get update
RUN apt-get install -y gconf-service libasound2 libatk1.0-0 libcairo2 libcups2 libfontconfig1 libgdk-pixbuf2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libxss1 fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils

# Install Chrome
RUN wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
RUN dpkg -i google-chrome-stable_current_amd64.deb; apt-get -fy install

# Install Python dependencies.
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

# Copy local code to the container image.
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . .

# Run the web service on container startup. Here we use the gunicorn
# webserver, with one worker process and 8 threads.
# For environments with multiple CPU cores, increase the number of workers
# to be equal to the cores available.
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 main:app

Configuring: `requirements.txt`

This Dockerfile uses a requirements.txt file with specific versions of all our Python dependencies. We'll need to install selenium as well as the specific version of the chromedriver-binary project that corresponds with the version of Chrome that we've installed:



# requirements.txt

Flask==1.0.2
gunicorn==19.9.0
selenium==3.141.0
chromedriver-binary==77.0.3865.40.0

Configuring: `main.py`

Finally, we'll write a Python application using Flask, Selenium



# main.py

from flask import Flask, send_file
from selenium import webdriver
import chromedriver_binary  # Adds chromedriver binary to path

app = Flask(__name__)

# The following options are required to make headless Chrome
# work in a Docker container
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("window-size=1024,768")
chrome_options.add_argument("--no-sandbox")

# Initialize a new browser
browser = webdriver.Chrome(chrome_options=chrome_options)


@app.route("/")
def hello_world():
    browser.get("https://www.google.com/search?q=headless+horseman&tbm=isch")
    browser.save_screenshot("spooky.png")
    return send_file("spooky.png")

Testing & Deploying

If we have Docker installed locally, we can run this to test it:



$ docker build -t my_screenshot_service .
$ docker run --rm -p 8080:8080 -e PORT=8080 my_screenshot_service

And view it at http://localhost:8080

Otherwise, we can deploy it directly to Cloud Run:



$ gcloud builds submit --tag gcr.io/YOUR_PROJECT/my_screenshot_service
$ gcloud beta run deploy my_screenshot_service --image gcr.io/YOUR_PROJECT/my_screenshot_service --region us-central1 --platform managed

And that's it!

A few notes:

We're using --no-sandbox to ensure compatibility with the Docker container, so only point such a service towards URLs you trust.
Be careful when exposing such a service to user input: For example, if the URL we were screenshotting was supplied by the user, they could potentially take a screenshot of any file on the filesystem as well!
Be sure to create a new service account with no permission and use it as the identity of the service, for better security. See https://cloud.google.com/run/docs/securing/service-identity for an example.

Developer Advocacy: Frequently Asked Questions

Dustin Ingram — Thu, 19 Sep 2019 15:34:46 +0000

I get a lot of questions about what it's like to be a Developer Advocate (DA). I love these questions because each one is an opportunity to advocate for developer advocacy itself!

Recently a colleague who's considering some DA positions reached out to me for advice, and their questions so nicely summed up every question I've ever been asked about my job that I figured I'd answer them here.

(Also, it ended up being way too long for an email).

A bit about me: I'm a Developer Advocate at Google, where I focus on advocacy for the Python community. I have a BS/MS in Computer Science, and spent many years working in university/government research labs until I decided to take a break from academia. After doing a number of non-software things, I joined a software consultancy where I spent a fair amount of time building custom software, before eventually joining Google.

Also, a disclaimer: I've only been a DA at Google. While I'm aware of differences between what this role means at other organizations, I really can only speak for what it means at my organization. I've also only ever been myself, so I also can't speak for all DAs at Google, either, but I don't think my experience is totally unique amongst us.

So, first, I'll answer a question which wasn't on the original list, but is one that I get asked the most:

What is a Developer Advocate?

The way I like to describe this is: I help represent the Python community at Google. When we build products that people in the Python community will use, I help make sure those products are right for those users.

This means that part of my job is to be deeply involved in the Python community. The way I do this is basically entirely up to me, so this usually manifests itself as the things I like doing in the Python community, like working on the Python Package Index (PyPI) or other packaging tools, organizing PyTexas, and attending and giving talks at Python conferences.

Being an advocate also means I don't want to "sell" you a product, but I do want you to know that it exists, but only if I think it might help you. I really don't want to show you tools you don't actually need, and I'm far more likely to tell you to use a simpler or cheaper tool if I think it's a better fit, even if it's not one of our products!

More importantly, if you are using the products I work with, I care a lot: I want to know what you think about them, if you've tried them, what your first impressions are, what your challenges have been, etc. And, I want to be a resource for you in the future.

How did you feel about leaving a traditional engineering role?

I get this question a lot; in fact, one of the first things a friend said to me after I told them I was going to be working in developer relations was: "So does that mean you aren't allowed to code anymore?"

I actually consider developer/user advocacy a part of a traditional engineering role, just one that is often neglected by traditional engineering organizations. So, while this usually means "how do you feel about not coding all the time", in fact it's entirely up to me: if I wanted to, I could be coding as much as someone in a "traditional engineering" role.

At Google, we currently have two titles: "Developer Advocate" (DA), and "Developer Programs Engineer" (DPE). The stereotype for a DA would be that they spend their time working with content, talking to customers, and giving talks. Similarly, the stereotype for a DPE would be that they build and maintain our client libraries, write code samples, and work on our open source projects.

But the truth is that these two titles just represent the extremes of a gradient of a single role, "Developer Programs Engineer". This means that some folks who are DAs still write code, and some folks who are DPEs give talks at conferences.

And while each individual does have a given set of responsibilities, the "slider" between these two extremes is more or less up to us. If I wanted to do more engineering than I currently do, there are plenty of opportunities for me to do so, and finding that balance is more or less up to me.

Do you feel that if you chose to go back to 'traditional engineering' that it would be possible?

Given all the above, I'd say yes, not only do I feel like I could go back to a "traditional" engineering role, but I probably have an even better perspective now and would be an even better engineer than before.

Being a DA has given me the ability to connect with such a wide and diverse group of users, and it constantly makes me reëvaluate whether a 'good idea' is what I, the engineer want, or what the user actually wants.

Are you finding a good work/life balance? I know DAs tend to travel quite a bit.

Absolutely, but this is heavily influenced by the culture at Google. The nice part about my job is that, contrary to popular belief, travel is not actually a requirement to be a DA at Google. There are DAs that are constantly traveling, and there are DAs that only travel once or twice a year, and both are totally fine and will be successful in their careers.

This means that I travel exactly as much as I want to, and at any time I have the ability to adjust how much I'm traveling to suit my work/life balance.

What do you find to be the biggest challenges with a role such as this?

The biggest challenge: measuring your impact.

Measuring impact is challenging because it's hard to connect the outcomes we want to achieve with the things we do. Things like "developers have more confidence in our products than a year ago" is not only a hard metric to measure, but also hard to find the source of the shift even if we could measure it. Was it all the talks we gave? The YouTube content? The Guinness World Record?

This makes it hard to know what's worth working on, and also makes it hard to prove that you're doing a good job. View counts on blog posts are a decent proxy for popularity, but since "having a popular blog" isn't our goal, beyond that it's hard to quantify.

What does the day of a DA look like at Google? What do you spend your time doing?

The impression is that a DA spends all their time writing talks, traveling to conferences, and speaking. This makes sense, because the DAs you probably see the most are the ones that are doing these things, and you don't see them doing anything else.

A lot of people are surprised to discover that traveling and speaking is actually a very small part of my job, given the number of talks I give, and how often they see me at events. And, like I mentioned before, the main reason I do it is because it's one of my ways to be part of the community, and something I enjoy.

I think part of this perception comes from the conflation between an advocate and an evangelist. There is endless debate about what these terms mean in the context of developer relations, but at least in the context of Google, the difference is this: an evangelist wants users to know that a product exists; advocates want products to know that their users exist.

(In my opinion, good way to tell if you're looking at an evangelist position or an advocacy position: who would you report to? Is it an engineering director? Or marketing?)

For example, you might notice very few of my general conference talks are actually about our products. This is because my talks are simply a means for interacting with my community, not a venue to inform you about our products.

So, if speaking is actually optional for being a DA, and I do it optionally, but takes up a small amount of my time, what the heck am I doing? Well, a lot:

Talking to customers: When I travel to events, I usually try to set up some meetings with local developers that use Google Cloud, or try to meet some while I'm there. I also occasionally sit in on calls with customers, either to check in on how their experience is going, talk about a specific issue, or help them determine how best to use our products.

Talking to non-customers: I'm generally and genuinely curious what everyone's working on, and the tools they're using, and whether these tools are effective or not. This helps me build a general sense of what people expect to be able to do with our products, strengths and weaknesses in other products, and new trends in the industry.

Talking to our product teams: Usually this is around a launch of a new product or feature, but sometimes is to set general strategy for the Python community.

Being "user zero" for new products or features: This means playing with new things and seeing how intuitive they are, and whether there are any edge cases we haven't considered, but might be valid use cases for our users, and filing bug reports as necessary.

Keeping an eye on places where users might raise issues: I am constantly trolling StackOverflow, and have various ways to keep an eye on mediums like DEV, Twitter, Hacker News, etc. for people talking about their experience with Google Cloud, and then taking that feedback and turning it into actionable issues for our engineering teams.

Working in open source: This generally satisfies my desire to write code, and has the added benefit of keeping me involved in the community, and also giving back to it.

Creating content for Python developers: This may be anything from blog posts, to launch announcements, to YouTube videos, to podcasts, to documentation, to workshops/tutorials/codelabs, to stunts like this.

Setting strategy for the Python community: I help determine our presence and sponsorships at events, but also contribute to wider plans for the future of our products.

Serving as a general source of expertise at Google about Python and the it's community: One of the most fun parts of my job is that I get pulled into discussions with teams all across Google who want input about how to work with the Python ecosystem, target Python users, and make things work in a way that they would expect and appreciate.

Leading a team of people who work on Python advocacy: I'm not the only person at Google who cares about Python! In order to coordinate our efforts, I lead a team of people who do all the same things I do (and often, much more).

Do you have any tips or advice for someone considering this role?

1) Work on your empathy.

At the very least, start doing more advocacy at your current organization. Attend or observe some user experience studies of your product, or just sit down with a friend who's never used it and see if they can make it through your quickstart. I promise, it will be eye-opening.

2) Pick your community, and deeply understand it.

Maybe you've already done the first half, but have you done the second? You need to be the voice at your organization for a huge group of people, and you need to be able to accurately represent what they want.

Keep in mind that the class of people who make up your community are not "everyone who uses our product" or "everyone who uses our competitors product". That's not a community, it's a market segment.

3) Consider joining a software consultancy first.

To quote the person who's taught me a lot about doing developer relations in the last year:

// Detect dark theme var iframe = document.getElementById('tweet-1137127323780624384-19'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1137127323780624384&theme=dark" }

Obviously I'm biased too, because this was my route to my current role, but I totally agree.

As a consultant, if you're not doing "client advocacy" with your team, you're gonna build something the client doesn't want. And if you can't educate the client, they're not going to know how to use what you built. Either way: you're not gonna have that client very long.

Finally, another question not on the original list, but one I get asked a lot:

Are you hiring?

There are developer relations teams across just about every product area at Google, and we're always looking for new colleagues. Take a look at this job search query to find a role that might suit you, and feel free to reach out to me on Twitter (@di_codes) or in the comments if you have any questions that aren't answered here!

HTML templates with Google Cloud Functions

Dustin Ingram — Thu, 23 May 2019 21:01:29 +0000

A Google Cloud Function can be called in response to multiple types of events, and of the most useful events to use is an HTTP request. This allows you to make a GET, POST or other HTTP request to a URL provided by your Cloud Function, and have it run in response to the given request.

When you activate your Cloud Function via HTTP, you can also optionally return a response, much like a HTTP server would. In the simplest examples, this is usually just a string:

def hello(request):
    return "Hello, World!"

This creates an HTTP response that looks like this:

HTTP/1.0 200 OK
Content-Length: 13
Content-Type: text/html; charset=utf-8

Hello, World!

You'll notice here that the Content-Type is text/html, but the response is not HTML at all, it's just plaintext. What if we wanted to add some formatting?

Adding some formatting

Let's say we wanted to make part of our response bold. We could do this by adding some HTML tags:

def hello(request):
    return "Hello, <b>World!</b>"

Which would give us the following response:

HTTP/1.0 200 OK
Content-Length: 20
Content-Type: text/html; charset=utf-8

Hello, <b>World!</b>

This would probably render correctly in our browsers, but only because modern browsers have an amazing propensity for dealing with invalid HTML. To make this actually a valid HTML response, we would need to change our function to be:

def hello(request):
    return """
<!DOCTYPE html>
<head>
  <title>Saying Hello</title>
</head>
<body>
  Hello, <b>World!</b>
</body>
"""

This is starting to get a bit long, so perhaps we would make it a separate variable:

INDEX_TEMPLATE = """
<!DOCTYPE html>
<head>
  <title>Saying Hello</title>
</head>
<body>
  Hello, <b>World!</b>
</body>
"""

def hello(request):
    return INDEX_TEMPLATE

Passing variables and controlling flow

If we want to pass variables into our template, we could just do it with .format:

INDEX_TEMPLATE = """
<!DOCTYPE html>
<head>
  <title>Saying Hello</title>
</head>
<body>
  Hello, <b>{place}!</b>
</body>
"""

def hello(request):
    return INDEX_TEMPLATE.format(place="World")

What if we had multiple places we wanted to say hello to, on separate lines?

def hello(request):
    return INDEX_TEMPLATE.format(places=["World", "dev.to"])

At this point, our "homemade" template would start break down, and we'd need to start getting creative in order to make it continue to work.

Inline templates with Cloud Functions

Instead, let's use some existing tools. Since under the hood Cloud Functions is using Flask, this means that the render_template_string function is available to us. This lets us render a Jinja2 template in a similar way as our custom solution, but also lets us do loops, if statements, etc:

from flask import render_template_string

INDEX_TEMPLATE = """
<!DOCTYPE html>
<head>
  <title>Saying Hello</title>
</head>
<body>
  {% for place in places %}
  Hello, <b>{{ place }}!</b><br>
  {% endfor %}
</body>
"""

def hello(request):
    return render_template_string(
        INDEX_TEMPLATE, places=["World", "dev.to"]
    )

This would produce a response such as:

HTTP/1.0 200 OK
Content-Length: 140
Content-Type: text/html; charset=utf-8

<!DOCTYPE html>
<head>
  <title>Saying Hello</title>
</head>
<body>
  Hello, <b>World!</b><br>
  Hello, <b>dev.to!</b><br>
</body>

Templates directory

Finally, perhaps your template is quite long and you don't want it to exist in the same file as your function, you can put it in a directory called templates and use the render_template function instead.

Your source files should look something like this:

├── main.py
├── requirements.txt
└── templates
    └── hello.html

With hello.html being the same as our previous example:

<!DOCTYPE html>
<head>
  <title>Saying Hello</title>
</head>
<body>
  {% for place in places %}
  Hello, <b>{{ place }}!</b><br>
  {% endfor %}
</body>

And the function configured like this:

from flask import render_template

def hello(request):
    return render_template('hello.html', places=["World", "dev.to"])

And that's it! From here, you can use all of Jinja2's features that you normally would in any other template.

Hitting a Cloud Function when you submit a Google Form

Dustin Ingram — Wed, 15 May 2019 17:55:06 +0000

Google Forms is a great tool--easy to create, easy to use, etc. I was recently at a meetup talking to users about Google Cloud Platform and someone said they wished they could hook a Google Form up to a Google Cloud Function.

After thinking for a second, I thought "this must be possible". Although there's not usually a lot of direct interoperability between G Suite and Google Cloud, I knew that you could write an Apps Script trigger for most Google Docs, which would let you make HTTP requests, and that Cloud Functions can accept arbitrary HTTP requests as events, so there's no reason this shouldn't work.

And in fact, it totally works!

Writing your Cloud Function

There's really nothing special about the Cloud Function you'll create for this: it's just like any other Cloud Function that accepts a HTTP request. Let's assume we're going to be passing JSON back and forth as the payload:

The function would look something like this:



def form_trigger(request):
    payload = request.get_json(silent=True)
    print(f"Payload was: {payload}")
    return "OK"

This will get the JSON from the POST request, print it to our logs, and just return an "OK". Obviously you can do then whatever you want with the payload at this point: store it in your database, kick off a job, etc.

Deploy that function with gcloud functions deploy form_trigger --trigger-http --runtime python37 and move on to the next step.

Creating your Google Form

The Google Form you create will just be like any other form: you can have multi-part questions, multiple-choice questions, free-form questions, etc.

Creating a script for your form

Here's where we can start connecting the dots. First, literally select the three dots in the menu when you're editing your form:

From this menu, choose "Script editor" to get taken to the Apps Script editor for this form. This should give you a file named Code.gs with an empty function like so:



function myFunction() {

}

We're going to update it to be something like the following:



// Replace with the URL to your deployed Cloud Function
var url = "<YOUR CLOUD FUNCTION URL>"

// This function will be called when the form is submitted
function onSubmit(event) {

  // The event is a FormResponse object:
  // https://developers.google.com/apps-script/reference/forms/form-response
  var formResponse = event.response;

  // Gets all ItemResponses contained in the form response
  // https://developers.google.com/apps-script/reference/forms/form-response#getItemResponses()
  var itemResponses = formResponse.getItemResponses();

  // Gets the actual response strings from the array of ItemResponses
  var responses = itemResponses.map(function getResponse(e) { return e.getResponse(); });

  // Post the payload as JSON to our Cloud Function  
  UrlFetchApp.fetch(
    url,
    {
      "method": "post",
      "payload": JSON.stringify({
        "responses": responses
      })
    };
  );
}

Make sure to update the url variable with the full URL to your deployed Cloud Function.

Adding a trigger

Now, from the script editor, click on "Edit" > "Current project's triggers" . This will prompt you to give your project a name, and then take you to the G Suite Developer Hub, and show you all the triggers for your project (there should be none).

In the bottom right corner, select "+ Add Trigger" to add a new trigger:

The dialog should default to the onSubmit function you declared for your form, otherwise select it.

You should also be sure to change the "Select event type" field from "On open" to "On form submit".

This will create a popup window to allow Apps Script to view and modify your form.

Finally, save your trigger and it should appear in the list of triggers.

Conclusion

At this point, your form is fully connected to Cloud Functions! You can submit some test responses and you should see the responses appearing in the logs for your function.

From here, you can hook the function up to other services, or do things conditionally based on the responses, and you shouldn't need to write additional Apps Script to make it happen.

Moving your cron job to the cloud with Google Cloud Functions

Dustin Ingram — Fri, 03 May 2019 00:53:18 +0000

A cron job is a way to run a program on a server at a specified interval. This is often a small script that does some repeatable task, like collect metrics, check the health of a service, take a snapshot of some dataset, etc.

The cron utility has a highly customizable format, called a crontab file, which allows us to run essentially any command at whatever interval we want. You can have your script run as often as once a minute, and as infrequently as once a year.

Most often, a cron job takes some input, does a little processing, and then generates some output. The input could be a web page, another application, a database, etc. The output might be anything from adding a row to a table in a database, to putting a file in storage, to sending a message.

Using cron jobs

Cron jobs have a lot of uses: essentially anything you want to happen in a repeatable way can be made into a cron job.

One thing I like to use cron jobs for is to keep an eye on websites that I visit infrequently. For example, I'd like to see articles posted to Hacker News about Serverless, but I don't have time to check it every day, and look at the title of every post to see if it's about Python.

Instead, I can write a cron job to do this for me. The steps will be roughly as follows:

Once a day, get the top stories from the API;
Iterate over every story;
If any of their titles match "serverless":
- send an email with the links

Here's the script:

#!/usr/bin/python

import requests
from utils import send_email

api_url = 'https://hacker-news.firebaseio.com/v0/'
top_stories_url = api_url + 'topstories.json'
item_url = api_url + 'item/{}.json'

def scan_hacker_news()
    # Make a request to the API
    top_stories = requests.get(top_stories_url).json()
    serverless_stories = []

    # Iterate over every story
    for story_id in top_stories:
        story = requests.get(item_url.format(story_id)).json()
        if 'serverless' in story['title'].lower():
            serverless_stories.append(story)

    # If there are any, send an email
    if serverless_stories:
        send_email(serverless_stories)

This makes a request to the API, iterates over every story, filters out stories that have "serverless" in the title, and sends me an email (we'll leave that function as an exercise to the reader).

If I wanted to set this up as a cron job on my Linux machine, I would save it to a file (send_me_pythons.py), make it executable (chmod 755 send_me_pythons.py) and put it in my local bin (/usr/local/bin/send_me_pythons.py).

Then, I would edit my crontab (/etc/crontab) and add the following line:

0 0 * * * /usr/local/bin/send_me_pythons.py

This runs the script once a day at midnight. I like to use https://crontab.guru/ to ensure that my crontab is right before I set it. Here's a diagram of each field:

# 0 0 * * * /usr/local/bin/send_me_pythons.py
# │ │ │ │ │ └── run this script
# │ │ │ │ └──── every year
# │ │ │ └────── every month
# │ │ └──────── every day
# │ └────────── at hour zero
# └──────────── at minute zero

Once the crontab is set, my machine will pick up on the new cron job and run it as often as I've specified.

Moving our script to the cloud

This is great, but it has one huge downside: now I have to keep my server up and running to ensure that the emails get sent, otherwise I'll be missing valuable Serverless content. And this is further complicated by the fact that either I should be using this server for other things besides running this little script once a day, or I need to keep an entire server up and running just for this little script! Not ideal.

Instead, let's take this function to the cloud. First, we'll turn it into a Google Cloud Function. We'll wrap our entire existing script in a Python function, and put it in a file called main.py:

import requests
from utils import send_email

api_url = 'https://hacker-news.firebaseio.com/v0/'
top_stories_url = api_url + 'topstories.json'
item_url = api_url + 'item/{}.json'

def send_pythons(request):
    # Make a request to the API
    top_stories = requests.get(top_stories_url).json()
    serverless_stories = []

    # Iterate over every story
    for story_id in top_stories:
        story = requests.get(item_url.format(story_id)).json()
        if 'serverless' in story['title'].lower():
            serverless_stories.append(story)

    # If there are any, send an email
    if serverless_stories:
        send_email(serverless_stories)

Note: Only the lines that need to happen every time the function is called actually need to be in the send_pythons function. The import statements only need to be executed once when the function is loaded, and can be left outside the function.

Next, we'll define our dependencies. We're using requests, so we need to put it in our requirements.txt:

requests==2.20.0

Then, we can deploy this with the gcloud command line tool:

$ gcloud beta functions deploy test --runtime python37 --trigger-http

This will give us an endpoint, something like:

https://us-central1-[PROJECT_ID].cloudfunctions.net/send_pythons

And making an HTTP GET request to that endpoint will result in our function being run.

Scheduling our script in the cloud

Now we've got our script as a function in the cloud, all we need to do is schedule it. We can create a new Google Cloud Scheduler job with the gcloud command line tool:

$ gcloud beta scheduler jobs create http send_pythons_job \
    --schedule="0 0 * * *" \
    --uri=https://us-central1-[PROJECT_ID].cloudfunctions.net/send_pythons

This specifies a name for our job, send_pythons_job, which is unique per-project. It also specifies the crontab schedule we set above, and points the job to our HTTP function we created.

We can list our job:

$ gcloud beta scheduler jobs list --project=$PROJECT
ID                LOCATION     SCHEDULE (TZ)        TARGET_TYPE  STATE
send_pythons_job  us-central1  0 0 * * * (Etc/UTC)  HTTP         ENABLED

And if we want to run our job out of schedule, we can do it from the command line:

$ gcloud beta scheduler jobs run send_pythons_job

Next steps

There's lots more you can do with Cloud Functions + Cloud Scheduler! Follow the links below to learn how to:

Schedule more complex tasks with Cloud Scheduler
Use Cloud Scheduler and Pub/Sub to trigger a Cloud Function
Send email from App Engine with Mailjet or SendGrid

All code © Google w/ Apache 2 license

Using Secrets in Google Cloud Functions

Dustin Ingram — Thu, 25 Apr 2019 20:41:46 +0000

Google Cloud Functions makes it easy to build serverless Python programs. This post will show you how you can use the Google Secret Manager to safely and securely use secrets in your function.

Hard-coding or using environment variables to store plain-text strings that should be "secret", like API keys, secret tokens for cookies, etc. generally isn't recommended. Any third-party dependency or library you use has access to these same environment variables.

However, it's very convenient to store secrets along side your function. You can still do it, as long as you store these secrets safely and securely.

We'll do this by storing our secret with the Google Secret Manager, and accessing our secrets at the application layer. Doing so will limit access to the secret to just members of your team who have access to the secret (and, of course, the function when it's running on Google Cloud).

Setup

I'll assume that you've already created a Google Cloud project with a name (mine is my_cloud_project). If you haven't, go ahead and do that now.

Storing your secret

Now that you've created a project, the next step is to create a secret in it. We can do this on the command line with the gcloud tool! Also, we'll do it via Cloud Shell, so no credentials ever have to leave the cloud.

First, launch Cloud Shell for your project.

You should see a prompt that looks like this:

Welcome to Cloud Shell! Type "help" to get started.
Your Cloud Platform project in this session is set to my_cloud_project.
Use “gcloud config set project [PROJECT_ID]” to change to a different project.
di@cloudshell:~ (my_cloud_project)$

Next, we'll enable the Secret Manager API (and the Cloud Functions API, while we're at it):

$ gcloud services enable secretmanager.googleapis.com cloudfunctions.googleapis.com

Next, we'll create a new secret with the gcloud command line tool:

$ echo -n "The chickens are in the hayloft." | \
    gcloud secrets create my-secret \
      --data-file=- \
      --replication-policy automatic

The --data-file=- flag allows us to pipe the secret to the gcloud command from the output of the previous command, which just echos the secret (with no newline).

Our secret is now securely stored. If we want to verify this, we can use the following command to access our secret on the command line:

$ gcloud secrets versions access 1 --secret="my-secret"
The chickens are in the hayloft.

Using your secret

Now we'll write our function. Here's our whole function that we'll deploy, that will live in a file called main.py:

import os
from google.cloud import secretmanager

client = secretmanager.SecretManagerServiceClient()
secret_name = "my-secret"
project_id = "my-gcp-project"
request = {"name": f"projects/{project_id}/secrets/{secret_name}/versions/latest"}
response = client.access_secret_version(request)
secret_string = response.payload.data.decode("UTF-8")


def secret_hello(request):
    return secret_string

This function is determining the resource name for the secret based on the function's environment, accessing the latest version of our secret, and then decoding the response, and finally returning the original string.

Also note how the function is doing this work outside of the Python function itself. This will speed up our function by only accessing the secret once when the Cloud Function is instantiated, and not once for every single request.

We'll also need to tell our function to install the google-cloud-secret-manager library, so we can access the secret from Python. In a file called requirements.txt, add the line:

google-cloud-secret-manager==2.0.0

At this point, your directory structure should look like this:

.
├── main.py
└── requirements.txt

Applying Permissions

We'll also need to give our function permission to access the secret:

$ gcloud secrets add-iam-policy-binding my-secret \
    --role roles/secretmanager.secretAccessor \
    --member serviceAccount:my-cloud-project@appspot.gserviceaccount.com

The serviceAccount here is the default service account for Cloud Functions.

Deploying your app

The last step is to deploy our function with the gcloud tool:

$ gcloud functions deploy secret_hello \
    --runtime python38 \
    --trigger-http

Now we can test our function once it's deployed:

$ gcloud functions call secret_hello
executionId: 4856xc8an4fa
result: The chickens are in the hayloft.

Next steps

There's lots more you can do with Cloud Functions! Follow the links below to learn how to:

Serverless Python Quickstart with Google Cloud Functions

Dustin Ingram — Thu, 18 Apr 2019 20:06:40 +0000

There's a new runtime in town: the Google Cloud team has brought everything awesome about Python to Google Cloud Functions by releasing a Python 3.7 runtime in addition to the existing Node.js runtime.

Cloud Functions are great for writing lightweight, self-contained microapps that can be individually scaled (and are a fraction of the cost of deploying a full webapp). They can also be used as "triggers": for example, automatically resizing an image into a smaller thumbnail once it's uploaded to a Google Cloud Storage bucket.

And now that you can write Cloud Functions in Python, you have the full power of the Python language (and any package from PyPI) at your disposal.

Initial setup

Cloud Functions have a simple directory structure:

.
├── main.py
└── requirements.txt

In main.py, we'll define our function. In requirements.txt, we'll add any requirements that we might need to install from PyPI (or just leave it empty if there aren't any).

Writing our function

To start, let's write a simple function in our main.py file that just says hello:

def hello(request):
    return "Hello World!"

Running locally

To quickly test out our function, we can run it locally before deploying.

You'll need Python installed to follow this part. If you don't yet, see "Setting Up A Python Development Environment".

Once you've got a Python environment locally, we can add the following code to the end of our main.py file:

if __name__ == '__main__':
    from flask import Flask, request
    app = Flask(__name__)
    app.route('/')(lambda: hello(request))
    app.run()

Next, create a virtual environment, activate it, and install Flask:

$ python -m venv env
$ source env/bin/activate
(env) $ pip install flask
Collecting flask
  ...
Successfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7
flask-1.0.2 itsdangerous-0.24

Run it as follows:

$ python main.py

...and visit http://localhost:5000 to see your endpoint locally.

Deploying our function

Our function is only useful if it's deployed! Before deploying our function, you'll need to install and initialize the gcloud command line tool.

Once the gcloud SDK is installed, we can deploy our function with one line:

$ gcloud beta functions deploy hello --trigger-http --runtime python37 --project YOUR_PROJECT_NAME

Once deployed, this command will give you a link to your live function.

Passing in variables

You're probably going to want to pass some data into your function. One way to do this is via URL parameters, like http://foo.bar/hello?name=Dustin. Let's make our example optionally take a parameter called name and use that instead of "World":

def hello(request):
    name = request.args.get('name', 'World')
    return f"Hello {name}!"

Then when we redeploying our function, and visit https://YOUR_PROJECT_NAME.cloudfunctions.net/hello?name=dev.to, it will say:

Hello dev.to!

Adding dependencies

One of the nice things about Cloud Functions is that you can install the same dependencies from PyPI that you would normally use anywhere else.

Let's add a dependency on the emoji package, so we can add an optional emoji to the end of our message as well.

We'll add the name and version of the package to our requirements.txt file:

emoji==0.5.1

This will automatically get installed when we redeploy the function.

Next, we'll want to import and use the library in our function:

import emoji

def hello(request):
    name = request.args.get('name', 'World')
    reaction = emoji.emojize(request.args.get('reaction', '!'))
    return f"Hello {name} {reaction}"

Then we can visit https://YOUR_PROJECT_NAME.cloudfunctions.net/hello?name=dev.to&reaction=:wave: and it will say:

Hello dev.to 👋

Next steps

There's lots more you can do with Cloud Functions! Follow the links below to learn how to:

Ministry of Silly Runtimes: Vintage Python on Cloud Run

Dustin Ingram — Tue, 09 Apr 2019 16:06:57 +0000

Introducing Cloud Run

Today, Google announced Cloud Run, a product that is really beautiful in its simplicity:

1) You write an service/app/function/whatever that listens for HTTP requests on $PORT;
2) You write a Dockerfile that inherits from any base image, installs any necessary dependencies, and starts your service;
3) You deploy the image to Cloud Run, and it scales on demand (even to zero).

Basically, it's "serverless Docker containers as a service" and it's exactly what I think the PaaS/FaaS development community could use right now, for a number of reasons:

The most idiomatic runtime possible
There's no potential for vendor-specific weirdness here. Really, you can't get more idiomatic than using FROM python:3.7, or whatever other base image you want to use.

Customizability without paying for idle virtual machines
If you have specific workloads that don't fit within the bounds of the most "common" runtimes usually provided, before now your best option was to spin up a custom VM which won't scale to zero and will spend a lot of time sitting idle. With Cloud Run, you can use the Dockerfile to customize your runtime to your heart's content.

Containerization means portability
Defining your runtime with a standard Dockerfile means you can take it anywhere—even locally—which means that you're not locked into a specific vendor, and that local development of your application is built right in without needing to do anything special.

These are all great, and they'll certainly help developers do great things...

...But what if you wanted to do something really silly with it instead?

Introducing the first Python 1.x serverless runtime

When I first started playing with Cloud Run, I really wanted to test the limits of what I could do with it. Could I really run any runtime I wanted? Could I run a super modern Python 3.8.0a3 alpha release? (Yes). Well, what about a "vintage" version of Python? Could I deploy a "Hello World" app running on Python 1.x to Cloud run? 🤔

Searching for "vintage" images

The first problem was finding a base image for Python 1.x. The official Python Docker images only publish images for 2.7 and up, so no luck there. I couldn't find it anywhere else, so it looked like I'd have to build the base image from scratch myself.

Searching for "vintage" source

Building the base image from scratch meant that I had to:

a) Find the old source code for a Python 1.x version;
b) Get it to compile on a modern distribution's base image;
c) Publish this new base image.

Finding the source was slightly challenging. The source for modern Python releases is published on https://www.python.org/downloads/source/, but this only goes back as far as Python 2.0.1, released June 22, 2001.

Luckily, I found https://legacy.python.org/download/releases/src/, which includes source releases as far back as Python 1.0.1, which was released just over 25 years ago on February 15th, 1994.

I decided to attempt to target the latest Ubuntu distribution, and amazingly, with just some small patches, I was able to get Python 1.0.1, 1.1, 1.2 and 1.3 to compile under Ubuntu 18.04.

Publishing some "vintage" images

I published these resulting builds as Docker images under the vintage-python repository, so this means you can now start a vintage Python REPL whenever you want:

$ docker run -it dustingram/vintage-python:1.0.1
Python 1.0.1 (Feb 23 2019)
Copyright 1991-1994 Stichting Mathematisch Centrum, Amsterdam
>>> print "Hello world!"
Hello world!

It also means I can do FROM dustingram/vintage-python:1.0.1 in my Dockerfile, which is a huge step towards having my silly runtime.

Searching for an HTTP server

I had my heart set on Python 1.0.1, but after realizing that Python didn't ship with an HTTP server until Python 1.3, I decided to use that instead of attempting to backport the HTTP server to an older version.

In Python 1.3, I was able to write this:

import sys
from BaseHTTPServer import HTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler

class MyHandler(SimpleHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        self.wfile.write("Hello from Python " + sys.version)


if __name__ == '__main__':
    print('Running...')
    HTTPServer(('0.0.0.0', 8080), MyHandler).serve_forever()

This sets up a basic HTTP/1.1 server on port 8080 that responds to every request with the given message. I really wanted to prove that this was a vintage version of Python serving the request, so I included the sys.version along with the response.

Containerizing some vintage Python

After saving that as app.py, all that was left to do was write the Dockerfile. Since my app doesn't have any dependencies, the Dockerfile is pretty close to the Dockerfile in the Cloud Run quickstart

FROM dustingram/vintage-python:1.3

# Copy local code to the container image.
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . .

# Service must listen to $PORT environment variable.
# This default value facilitates local development.
ENV PORT 8080

# Run the web service on container startup.
CMD ["python", "app.py"]

Here, we inherit from my vintage Python image, copy in my app.py to the container, set the PORT environment variable (for local development), and then start the server by running python app.py.

Deploying some vintage Python

With Cloud Run, there's a two step deploy:

Build and submit your image with Cloud Build:

$ gcloud builds submit --tag gcr.io/my_project_id/helloworld

Then, deploy the submitted image to Cloud Run:

$ gcloud beta run deploy --image gcr.io/my_project_id/helloworld

And then it was deployed!

See it in action here: https://helloworld-bpzdnjbrsq-uc.a.run.app/

Conclusion

Cloud Run is definitely flexible enough to run any runtime I could throw at it -- even one of the oldest versions of Python. Obviously I wouldn't seriously encourage anyone to use Python 1.x in production: any version past it's EOL is essentially unmaintained and severely feature-limited.

From here, you can:

See the vintage-python Github repo for more details on the runtimes;
Use the vintage-python images in your own silly project;
See the Cloud Run quickstart;
Follow me on Twitter.

DEV Community: Dustin Ingram

Portable Cloud Functions with the Python Functions Framework

What are the Functions Frameworks?

Going beyond Cloud Functions

An example: Portability without the Functions Framework

An example: Portability with the Functions Framework

A local development experience

Bonus: Static Typing

Next steps

Using Headless Chrome with Cloud Run

Configuring: Dockerfile

Configuring: requirements.txt

Configuring: main.py

Testing & Deploying

Developer Advocacy: Frequently Asked Questions

What is a Developer Advocate?

How did you feel about leaving a traditional engineering role?

Do you feel that if you chose to go back to 'traditional engineering' that it would be possible?

Are you finding a good work/life balance? I know DAs tend to travel quite a bit.

What do you find to be the biggest challenges with a role such as this?

What does the day of a DA look like at Google? What do you spend your time doing?

Do you have any tips or advice for someone considering this role?

Are you hiring?

HTML templates with Google Cloud Functions

Adding some formatting

Passing variables and controlling flow

Inline templates with Cloud Functions

Templates directory

Hitting a Cloud Function when you submit a Google Form

Writing your Cloud Function

Creating your Google Form

Creating a script for your form

Adding a trigger

Conclusion

Moving your cron job to the cloud with Google Cloud Functions

Using cron jobs

Moving our script to the cloud

Scheduling our script in the cloud

Next steps

Using Secrets in Google Cloud Functions

Setup

Storing your secret

Using your secret

Applying Permissions

Deploying your app

Next steps

Serverless Python Quickstart with Google Cloud Functions

Initial setup

Writing our function

Running locally

Deploying our function

Passing in variables

Adding dependencies

Next steps

Ministry of Silly Runtimes: Vintage Python on Cloud Run

Introducing Cloud Run

Introducing the first Python 1.x serverless runtime

Searching for "vintage" images

Searching for "vintage" source

Publishing some "vintage" images

Searching for an HTTP server

Containerizing some vintage Python

Deploying some vintage Python

Conclusion

Configuring: `Dockerfile`

Configuring: `requirements.txt`

Configuring: `main.py`