DEV Community: Eli Holderness

Using Coiled with Anvil

Eli Holderness — Tue, 21 Feb 2023 14:52:25 +0000

If you've got some big datasets and you want to wrangle some insights from them, you might want to distribute all that heavy-duty computation. Coiled lets you do just that, and takes the hosting off your plate too. So, of course, I built an app to showcase just how easy it is to use Coiled with Anvil. Today's blog post is a tour of that app and how it works, and it aims to give readers a good idea of how they might extend the app to meet their own needs.

Here's what the finished app looks like when it's first loaded. Later, it'll display output data.

This example app uses Coiled to do some computation on a randomly generated Dask timeseries dataset, then generates Plotly plots from that output data to display to the user.

It also shows how to use Anvil's PDF generation to provide a PDF version of that output, for the user to download.

By the end, you'll know how to use Coiled with Anvil, and be able to build an app of your own!

Read on to learn more about the following:

You can also clone the app to read the code and follow along.

Let's dive in!

What is Coiled?

Coiled provides a platform for users to spin up clusters for for distributed computation using the popular Python framework, Dask.

All you have to do is to use it is create an account and link that up an existing AWS or GCP account (so that Coiled knows where to create your clusters!). From there, it's all Python.

For the purposes of this blog post, I'll be assuming a basic familiarity with Coiled, but all you really need to know is that Coiled is a fast and easy way to do some serious computation - without needing access to any powerful computers yourself. You can see some great examples of Coiled usage on their blog.

What is Anvil?

Anvil is a platform for building web applications using nothing but Python, which means we need to know a little about the web. In particular, web apps have front-end code, also called client code, (which runs in the user's browser) and back-end code (which runs on a server somewhere). Anvil lets you write both front-end and back-end code in Python, and communicate by calling back-end functions from front-end code.

Anvil also has a drag-and-drop Editor for building user interfaces, which we'll be using later in this walkthrough.

Learn more about front ends, back ends, and how web applications work in Anvil!.

How can we use the two together?

You can install any custom package you like into your Anvil apps' server environments, so we'll be driving Coiled from back end code.

Coiled is typically used in notebooks such as Jupyter notebooks, or Google Colab notebooks. Anvil's server environments differ slightly from these environments in a couple of ways that are important here:

When client code in an Anvil app makes a call to the server, the server environment is created anew each time (unless you're using the Persistent Server for your Anvil app)
Normal server calls on Anvil apps have a time-out of 30 seconds, meaning that long-running tasks (such as running a Coiled computation) need to be run as Anvil Background Tasks.

The first point means that we need to make sure that all the necessary configuration to connect to Coiled is in place every time the server environment loads.

The second point means that for any long-running function calls, we'll be using Anvil's Background Tasks, which are specifically designed to handle situations where a call will take a while to complete.

With these two things in mind, we're ready to get building. Since Coiled is a Python package, we can install it into any Anvil app with access to custom packages, and we're off!

Configuring your Anvil app's server environment

Whenever our server environment loads, we'll need it to be correctly configured so that it can connect to Coiled. To do this, we can set the details we need from code using dask.config.set, like this:

dask.config.set({
  "coiled.user": "my-coiled-username",
  "coiled.server": "https://cloud.coiled.io"
})

python

Adding a dask.config.set call at the top of our Server Module will ensure that any code that runs in our server environment will be doing so with this configuration in place.

For this example app, we'll be configuring the following:

The name of the Coiled account it should connect to
The name of the specific user within that Coiled account
An API token associated with this user
The server that should be used to provision clusters

More information on how to find each of these pieces information can be found in the Coiled documentation.

If you're used to a notebook environment, you might be familiar with using a config.yaml file to configure your Coiled environment. You could use the approach detailed in this blog post to set any of the options you'd typically set in a YAML configuration file.

The account name, username and server aren't secret, but the API token should be stored securely. To do this, we'll use Anvil's App Secrets so that the token is encrypted at rest, and leave the other configuration details hard-coded.

That means that we can write a function to configure our Python environment to use Coiled correctly, and that code looks this:

import dask
dask.config.set({
  "coiled.token": anvil.secrets.get_secret('coiled_token'), # get the API token
  "coiled.user": "eli-holderness", # that's me!
  "coiled.account": "eli-holderness",
  "coiled.server": "https://cloud.coiled.io",
})

Putting this code at the top of our Server Module means that it'll always run before anything else, which is exactly the behaviour we want.

Using Background Tasks

When we actually use Coiled, we'll be doing it from inside a Background Task, so that our code has the time it needs to run. These tasks are written in our app's server code, and we'll define two Background Task functions there - one for spinning up a new cluster, and one for doing all the computation.

For example, here's the code for spinning up a new cluster from a Background Task:

DEFAULT_CLUSTER_NAME = "anvil-coiled-demo"

...

@anvil.server.background_task 
def setup_cluster(name):
  setup_config()
  cluster = coiled.Cluster(
    name=name if name else DEFAULT_CLUSTER_NAME,
    n_workers=1,
    scheduler_options={"idle_timeout": "1 hour"},
    shutdown_on_close=False,
  )

We can see that it takes a name variable as input. When we build our UI, we'll provide a way for the app's user to choose a name for the new cluster, with a default option hard-coded within the Server Module.

The way to launch a Background Task is from another server function:

@anvil.server.callable
def start_cluster(name):
  task = anvil.server.launch_background_task('setup_cluster', name)
  return task

We can see that when the Background Task is launched, a task variable is returned, which provides the mechanism for callbacks to the task. From within that task's function, we can write to the task's state (a dict-like object) with anvil.server.task_state['key'] = value, and from outside the function we can read that state by calling task.get_state(). For more detail, see Communicating with Background Tasks.

Later, we'll be using task.get_state() to retrieve the results of our computation, and task.is_completed() (which returns True if the function has finished, and False otherwise) to check on the status of our long-running tasks from client code.

Building a UI

Next, we're going to build the UI for our app, all in Python, with Anvil's visual UI editor. With the Editor, you can drag and drop visual components onto your app, and then interact with these components using client-side Python code. The pages for your app are called Forms, and the app we're going to build will have two Forms - a StartupForm and a ResultsForm.

The app we're building today is actually very simple. It spins up a cluster, connects to that cluster to do some basic computation, then displays the outputs.

So, we're going to build the following:

one Form (the StartupForm) that lets the user choose a name for their new cluster and spin it up, then run the computation
one Form (the ResultsForm) that displays the results of that computation

The StartupForm has a few components: A TextBox that asks for a name for the new cluster, two Buttons, and some Label elements that display text, including the title of the app.

Here's how the StartupForm for our app looks in the Anvil Editor:

Not every element will be visible at once; in fact, when the Form is loaded, its initialisation sets some of the UI elements to be either disabled or invisible.

At the very bottom, there's a Timer element. This is an invisible UI element that can regularly and repeatedly execute code, which makes it perfect for checking on the state of any Background Tasks we launch.

Here's the initialisation code for the Form, in its __init__ method:

  def __init__(self, **properties):
    # Set Form properties and Data Bindings.
    self.init_components(**properties)

    # Any code you write here will run when the form opens.
    self.run_calculations_button.enabled = False
    self.spinning_up_text.visible = False

    self.setup_task_running = False
    self.plot_task_running = False
    self.timer_1.interval = 0

When the Form is loaded, the button to run calculations is disabled, and the text that tells us our cluster is being provisioned is hidden. Later, we'll use code to make that text visible and to enable that button at the appropriate points during the user flow. The Timer element also has its interval property set to zero, which disables it.

To start with, the user optionally enters a name into the cluster_name_box (the text box at the top of the Form), and then clicks the button that says 'Spin up a new cluster'. We need this button to make a call to a server function that will spin up a cluster for us, with the appropriate name.

Here's the Python function that will run in the browser when that button is clicked. It calls a server function, and changes some visual attributes of the UI:

  def spin_up_button_click(self, **event_args):
    """This method is called when the button is clicked"""
    self.cluster_name = self.cluster_name_box.text
    with anvil.server.no_loading_indicator:
      self.setup_task = anvil.server.call('start_cluster', self.cluster_name)
    self.setup_task_running = True
    self.spin_up_button.enabled = False
    self.spinning_up_text.visible = True
    self.timer_1.interval = 1

We can see that the Form stores the text contents of the cluster_name_box as an attribute on itself (with self.cluster_name = ...), and then also passes that name on to the function that will launch our first Background Task.

Next, there's some state on the client side that needs to be updated; we know that the setup task is now running, so we can disable the button that launches it (that's the button the user just clicked). We can also show the spinning_up_text label, so that the user knows the task is in progress.

Finally, we enable the Timer (with self.timer_1.interval = 1), so that we can start checking every second to see whether our cluster has finished spinning up.

If you wanted to extend this app, you could add more UI elements to allow the user to set other parameters for their new cluster - for example, n_workers or scheduler_options.

Once the cluster has spun up, the Background Task that handles it will be completed. We can check for this using our Timer component. Every time our Timer ticks, we can inspect that Background Task objects and see if it's done yet.

Here's some of the code that runs every time the Timer ticks:

  def timer_1_tick(self, **event_args):
    """This method is called every [interval] seconds. Does not trigger if [interval] is 0."""
    # Check whether the setup task has completed
    if self.setup_task_running and self.setup_task.is_completed():
      self.setup_task_running = False
      self.run_calculations_button.enabled = True
      self.timer_1.interval = 0
      alert("Your new cluster is ready!")

This code checks on the status of the setup_task, and - if it's complete - the button to run calculations will be enabled. The Timer disables itself by setting its interval to 0, and sends an alert (a pop-up) to give the user a very clear visual sign that they can now run their calculations.

The button that says 'Run calculations' is now enabled. When the user clicks it, we want to run the code that will connect to our new cluster and do the computation we want. Here's the event handler for the 'Run calculations' button:

  def run_calculations_button_click(self, **event_args):
    """This method is called when the button is clicked"""  
    with anvil.server.no_loading_indicator:
      self.plot_task = anvil.server.call('execute', self.cluster_name)
    self.plot_task_running = True
    self.timer_1.interval = 1

Just as before, the event handler calls a server function to launch our second Background Task, and enables the Timer element. The Timer element should now also check for the status of the second task, so we'll add the following code to its event handler:

  def timer_1_tick(self, **event_args):
    """This method is called Every [interval] seconds. Does not trigger if [interval] is 0."""
    # Check whether the setup task has completed
    if self.setup_task_running and self.setup_task.is_completed():
      ... # this is the code in the code block two sections above this one

    # Check whether the plotting task has completed
    if self.plot_task_running and self.plot_task.is_completed():
      self.timer_1.interval = 0
      open_form('ResultsForm', self.plot_task.get_state()['row'])

Once the second Background Task is complete, we'll have some data to plot, so the Timer accesses that data with self.plot_task.get_state()['row'] and opens the ResultsForm to display it. We'll take a look at the second Form later - but first, we'll need to generate that output data! Let's take a closer look inside that second Background Task.

Wrangling some data

This is the part that'll be familiar to anyone used to working with data: some computation. This app is just a demo, so we'll use Dask's built-in timeseries datasets. This function generates a dataset with one name per second over a set period of time, and using it looks like this:

ddf = dask.datasets.timeseries(
    start="2000-01-01",
    end="2000-01-02",
    freq="5s",
    seed=42,
  )

Once that dataset exists, we use it to create two sets of output showing how many times each name appears in the dataset. For the first plot, we use the whole dataset, and for the second we take a random sample of 0.5% of the dataset.

Once we've got each of those sets of output - that is, some x-values and y-values - we'll store them into a Data Table, called output. It's best practice not to leave large amounts of data in the state of a Background Task, but we can store simple Python objects - like lists - in Simple Object columns in a Data Table. Then, we can retrieve that Data Table row from the Background Task's state once the task has completed.

Here's what the Data Table for storing our outputs looks like:

Here's the code for calculating and storing the frequencies of all the names in our dataset:

# create a new Data Table row to store our outputs
row = app_tables.output.add_row()

# get the frequency for each name
total_counts = ddf.groupby(ddf.name).count().drop(labels=["x", "y"], axis=1).compute()['id']

# store the x- and y-values as a list of lists in a Data Table
row['total_counts'] = [total_counts.index.values, total_counts.values]

# return this output in the Background Task's state
anvil.server.task_state['row'] = row

We can do the same for a random sample of 0.5% of our dataset. And with that, the data wrangling is done!

Plotting our data

Now that we've got some output from our data in the form of two sets of x- and y-values, we want to display them to the end user. To do this, we'll build a second Form called ResultsForm which will plot our output data, and display it to the user in the browser. This Form will also be used to create a PDF version of the output, so the user can download and keep it.

The ResultsForm is visually very simple - just two Plots, a Label for each, and some buttons at the bottom for the user to either download this page as a PDF, or to return to the previous Form.

On the left, you can see how the ResultsForm looks in the Anvil Editor, and on the right you can see it populated with bar plots.

This Form is loaded from the Timer element on our first form, the StartupForm, after the computation task finishes, and at this point the output data is passed through to the ResultsForm. So, when this new Form loads, we want it to use that output data to generate some plots, and we can write the code to do that in the ResultsForm's __init__ method.

Anvil has client-side integration with Plotly, which means we can use graph_objects to turn our output data into bar plots, and display those in the Form's Plot components.

Here's some of the code that runs in the ResultsForm's __init__ method:

  def __init__(self, row, ...):
    ...

    import plotly.graph_objects as go

    # get the latest set of output data from our Data Table
    total_plot_data = row['total_counts']
    self.total_plot.data = go.Bar(
        x = total_plot_data[0],
        y = total_plot_data[1]
      )

    random_sample_data = row['random_sample']
    self.random_plot.data =  go.Bar(
        x = random_sample_data[0],
        y = random_sample_data[1]
      )

This code uses the row variable that was passed in from the calling code in the StartupForm, and uses it to create bar plots which are displayed in the Plot components on the ResultsForm.

Generating PDF output

Anvil can render any Form as a PDF and make it available for user download. However, if we're rendering our ResultsForm as a PDF, we don't necessarily want all the UI elements on it - such as the buttons - to be included. We want to have conditional formatting for the Form, depending on how it's being rendered.

To make this happen, the ResultsForm has an optional variable in its __init__ method, as_pdf=False. We use this to optionally show various bits of the Form depending on whether it's being shown in the browser, or rendered as a PDF. This way, when we want to render the Form as a PDF, we can pass as_pdf=True, and the Form can use that flag to hide any extraneous UI elements:

  def __init__(self, row, as_pdf=False, **properties):

    ...

    if as_pdf:
      self.pdf_button.visible = False
      self.return_button.visible = False

When the Form is opened in the browser, as_pdf will always be False, so the buttons will show. When we create the PDF, we can set this variable to True, hiding the buttons.

Turning this Form into a PDF happens on the server side, in a function that looks like this:

@anvil.server.callable
def get_pdf(row):
  pdf = anvil.pdf.PDFRenderer(
    quality="original",
    page_size="A4",
    filename="Anvil Coiled Results.pdf"
  ).render_form(
    'ResultsForm',
    row, # pass in the output data so the Form can generate its plots
    True # sets the `as_pdf` variable to `True`
  )
  return pdf

That function is called from the 'Download PDF' button's event handler in the browser, and the PDF is then downloaded. Here's that event handler:

  def pdf_button_click(self, **event_args):
    """This method is called when the button is clicked"""
    pdf = anvil.server.call('get_pdf', self.row) # self.row = row was set in the Form's __init__ method
    anvil.media.download(pdf)

To learn more about creating and downloading PDFs with Anvil, check out our feature guide here.

Wrapping up

And that's it! In this walkthrough, we've seen how to build an app that integrates with Coiled to let a user spin up a cluster and run computations with it. We've also seen how to build a flexible, extensible UI in Anvil, so that we can share this functionality by publishing our app.

Going even further

This app is just a demo, showing the bare bones of how you can use Anvil to run Coiled code. In the real world, you'd be using a proper dataset, and likely allowing the end user to configure far more options for their clusters than just what's demonstrated here. You could have all sorts of different available computations, taking parameters from the users; you could make more use of Plotly's interactivity on the front-end, and you could use Anvil's built-in user authentication to restrict actions like cluster provisioning to certain users. You could even store configuration options in a Data Table, and allow users to choose between multiple different Coiled profiles.

If you do want to tinker with this for yourself, all you need to do is clone this app into your own Anvil account, set your configuration details, and put your own Coiled API token into your app's Secrets. Have fun!

More about Anvil

If you're new here, welcome! Anvil is a platform for building full-stack web apps with nothing but Python. No need to wrestle with JS, HTML, CSS, Python, SQL and all their frameworks – just build it all in Python.

Try Anvil - it's free, forever.

Using custom Python packages with Anvil

Eli Holderness — Thu, 24 Nov 2022 17:02:10 +0000

Anvil is a platform for building web apps using nothing but Python, both on the client and the server. This means that, with Anvil, you can create and deploy a web application without needing to learn HTML, JavaScript or CSS. Moreover, it means that Anvil developers can take advantage of Python's rich library of packages, and build a web interface that makes use of those packages.

If you're completely new to Anvil, you might like to check out our Feedback Form tutorial - it'll walk you through the essentials of building an Anvil app!

You can install and use any Python package you like in your Anvil apps, either directly from PyPI, or any method supported by pip's requirements.txt file method. Packages you install are available in your app's Server Modules, and can be used from client Python code in that app by making server calls.

This tutorial will walk you through the process of building an Anvil app that uses custom packages, and by the end you'll know how to install and use any custom package you want in your Anvil apps.

In this example tutorial, you'll build an app that generates and displays a word cloud based on the text of a random Wikipedia page. You'll use the wikipedia, wordcloud and Plotly Express packages to do this.

To use custom packages in your Anvil apps, you'll need a Personal plan or above. If you're on a Free plan, you can get an instant 7 day trial, no credit card required - just follow this tutorial, and hit the Start Trial button when prompted!

Here's what you'll build:

This tutorial contains the following sections:

Step 1: Create your Anvil app

First, you'll need to create your new app. To get started, head to the Anvil Editor. On the landing page of the Editor, click 'Create a new app'.

Choose 'Blank app' from this screen., and from the dialog that appears, choose 'Material Design'.

You'll now see the Anvil Editor.

You are now editing a new, blank Anvil app, and you're ready to start building!

Step 2: Install packages into your app's server environment

In order to use custom packages in your app's Server Modules, you'll need to install them into the environment that your server code runs in. This step shows you how to do that.

To install custom packages, go to your App's Settings page. Select the gear icon from the Sidebar Menu, and you'll see this page:

Now select 'Python versions' from the options on the left, and then select 'Python 3.10' from the drop-down.

Anvil provides several base environments, each of which has various packages pre-installed. You're going to start with the Data Science environment, then install three packages in addition.

From the 'Base packages' section, choose 'Data Science' from the drop-down. Then, under the 'Package' section, add the following packages:

wikipedia
wordcloud
plotly-express

For each package, type its name as it appears above into the text box on the left, and leave the right-hand text box blank. Then, click the 'Add' button that appears, and you will be able to add another package. Here's how it will look as you add packages:

Now, your packages are ready to use in your Anvil app's server environment.

Step 3: Write a function to generate word clouds

Now you've installed our libraries, it's time to write some code that uses them! Custom packages are available in Server Modules, so you're going to write a server-side function to generate your word cloud. Then, in Step 4, you'll build a client-side user interface to call that server function and display the results.

If you want to learn more about the structure of the web and the differences between Client and Server code, check out our Client vs Server Code in Anvil article.

To add a Server Module to your app, go to the App Browser by clicking the folder icon at the top of the Sidebar Menu. Then, under 'Server Code' on the left, click on Add Server Module

This is the Python environment in which you can write your server function. At the top, add the following lines to import the packages you installed earlier:

import wikipedia
from wordcloud import WordCloud
import plotly.express as px

Next, you're going to write a function that gets data from a random Wikipedia page, uses it to create a word cloud, and then returns that word cloud along with a link to the Wikipedia page.

Getting a random Wikipedia page

Here's the part where you'll write Python code to use your custom packages.

You can get a title of a random Wikipedia page with the wikipedia.random() function, as seen in the package's documentation. Once you have a title, you can get information about that page by calling the wikipedia.page() function.

So, to start, your function looks like this:

def get_cloud():
    title = wikipedia.random()
    page = wikipedia.page(title)
    text = page.content
    url = page.url

However, it's possible that the title returned by wikipedia.random() might give you a DisambiguationError if you call wikipedia.page() with it. This represents the 'Disambiguation' pages on Wikipedia (example). To avoid this, add code to retry the function call up to 5 times if that happens:

def get_cloud():
  for i in range(5):
    try:
      title = wikipedia.random()
      page = wikipedia.page(title)
      text = page.content
      url = page.url
    except wikipedia.exceptions.DisambiguationError:
      continue
    break

Next, you'll want to take that text and use it to create a word cloud. Do this with the WordCloud class you imported at the top of your Server Module.

    wordcloud = WordCloud(width=1000, height=700).generate(text)

python

Now, you'll use Plotly Express to turn that WordCloud object into something we can display.

  cloud = px.imshow(wordcloud)
  cloud.update_xaxes(showticklabels=False)
  cloud.update_yaxes(showticklabels=False)

Finally, this function needs to be able to be called from your app's client code, which we'll see in the next section. To make your function callable from the client, add the @anvil.server.callable decorator above the def line.

Your final function should now look like this.

@anvil.server.callable
def get_cloud():
  for i in range(5):
    try:
      title = wikipedia.random()
      page = wikipedia.page(title)
      text = page.content
      url = page.url
    except wikipedia.exceptions.DisambiguationError:
      continue
    break
  wordcloud = WordCloud(width=1000, height=700).generate(text)
  cloud = px.imshow(wordcloud)
  cloud.update_xaxes(showticklabels=False)
  cloud.update_yaxes(showticklabels=False)
  return cloud, title, url

Now, you're ready to build your app's user interface!

Step 4: Build your app's UI

In this step, you'll use Anvil's drag-and-drop UI designer to build a user interface for your app. Every component that you drag and drop onto your app's page is a Python object, with properties that you can set either from the designer itself, or by setting them in client code.

In the App Browser, click on 'Form1' under 'Client Code'. This will open up your app's Form within the Editor, allowing you to start building your UI.

On the right, you'll see the Component Toolbox. You can drag and drop visual elements from the Toolbox onto your app's Form.

To start, drag and drop a Label component into the Title slot on your Form. Click on it to bring up its Properties Panel, and change its text property to "Wikipedia Word Clouds".

Here's how that looks:

Similarly, you're going to want to add a few other components on your app's Form:

Drag a Card component into the main body of your Form.
Within that Card, drop a Plot component.
Drop a Link component below that, and change its alignment to center and its font size to 40.
Drag a Button component underneath your Card component, and change its text property to Give me a word cloud!, and its role to primary-color.

Here's how that looks:

Finally, set your Card's visible property to False by unchecking the box. Later, once we have a word cloud to display within it, we'll change this property from client code, making it visible again.

Now, you're done with dragging and dropping components. It's time to write some client code!

Step 5: Writing client Python code

In the last step, you built your UI. Now, you'll write some client-side Python code that will define how your UI behaves when a user interacts with it.

Double-click on the Button at the bottom of your Form. This will take you to the code that drives your app's Form.

A button_1_click method has just been added to your Form's code; this method is called whenever a user clicks on the button. Modify it so that it calls your server function, get_cloud, and uses the return values to change the UI elements on your Form:

  def button_1_click(self, **event_args):
    """This method is called when the button is clicked"""
    cloud, title, url = anvil.server.call('get_cloud')
    self.button_1.text = "Give me another word cloud!"
    self.card_1.visible = True
    self.plot_1.figure = cloud
    self.link_1.text = title
    self.link_1.url = url

First, this method gets the word cloud, title and URL for a random wikipedia page, using the server function you wrote earlier. Then, the method does the following:

changes the text on the Button
makes the Card (along with the Plot and Link) visible
changes the Plot to display the word cloud
changes the Link so that it links to the Wikipedia page, with the title of the page as its text property

And that's all the code you need to write! Click 'Run' in the top right of the Editor, click the button that says 'Give me a word cloud!', and watch your app generate a word cloud for you.

Finally, you can publish your app to make it available to anyone you like, by clicking the 'Publish' button in the top right.

Clone the App

You can click the link below to clone the example app, and see how yours compares:

Clone to your account

Happy building!

New to Anvil?

Yes – Python that runs in the browser. Python that runs on the server. Python that builds your UI. A drag-and-drop UI editor. We even have a built-in Python database, in case you don’t have your own.

Why not have a play with the app builder? It's free! Click here to get started:

Get building!

Build a Morse Code Generator with the Pico W

Eli Holderness — Fri, 07 Oct 2022 15:35:35 +0000

Flash Morse Code Messages on the Pico W - with an interactive, secure, authenticated web app!

This tutorial shows you how to create an Anvil app which takes a text input from a visitor, converts that text into Morse code, and makes your Pico W's LED flash with that Morse code message. It also shows you how to add user authentication to your app, so that you can control who has access to it.

It consists of the following steps:

Getting started

First, you'll need to get your Pico W running with the Anvil Uplink. Follow the Getting Started guide, then come back here when you're done!

Showing Morse code with your Pico's LED

For this Morse Code tutorial, you'll be using a bigger Python environment than your Pico W, (in your Anvil app itself), to do the Morse conversion. Then, you'll call from code there to code running on your Pico so that the Pico only has to render the dots and dashes.

Let's start with the code that will run on your Pico. Open up main.py from your Pico, which should appear as a drive on your computer after installing the Anvil firmware.

This is how your Pico W will appear after you've installed the Anvil firmware. Open up the main.py file. At the bottom of the file, you'll see this line of code:

anvil.pico.connect(UPLINK_KEY, no_led=True)

Directly above that, add the following code, which defines a function to make the LED on your Pico W blink fo a set duration:

# We use the LED to display the message in Morse Code
led = Pin("LED", Pin.OUT, value=0)


async def blink(duration_ms):
    led.on()
    await a.sleep_ms(duration_ms)
    led.off()

Now, add the following function, which takes a Morse code string input and makes the LED of your Pico blink appropriately:

@anvil.pico.callable_async()
async def morse_message(msg):
    print(f"{await anvil.pico.get_user_email()} has requested the message: '{msg}'")
    for m in msg:
        if m == ".":
            await blink(100)
        elif m == "-":
            await blink(400)
        else:
            await a.sleep_ms(700)
        await a.sleep_ms(200)

The anvil.pico.callable_async decorator means that this function will be callable from your Anvil app when your Pico is connected to the Uplink.

Now, save and close this file, and reboot your Pico W.

Translating text into Morse code

Next, you'll need to write the code that takes a text input and turns it into Morse code. This bit will be done in your Anvil app itself.

In your Anvil app, open up your app's Structure view by clicking on the top icon on the left-hand sidebar. In the menu that pops out to the right, add a Server Module by clicking on the three-dots menu next to 'Server Code', and clicking 'Add Server Module'.

This will open up your new Server Module in the Editor.

This is where you can write code that will run a full Python 3 environment, and it's where you're going to write the function that will turn text into Morse code.

To do this, your app will need a way to translate text characters into Morse characters, which is a perfect job for a dict. Add the following into your Server Module:

MORSE_DICT = { 'A':'.-', 'B':'-...',
               'C':'-.-.', 'D':'-..', 'E':'.',
               'F':'..-.', 'G':'--.', 'H':'....',
               'I':'..', 'J':'.---', 'K':'-.-',
               'L':'.-..', 'M':'--', 'N':'-.',
               'O':'---', 'P':'.--.', 'Q':'--.-',
               'R':'.-.', 'S':'...', 'T':'-',
               'U':'..-', 'V':'...-', 'W':'.--',
               'X':'-..-', 'Y':'-.--', 'Z':'--..',
               '1':'.----', '2':'..---', '3':'...--',
               '4':'....-', '5':'.....', '6':'-....',
               '7':'--...', '8':'---..', '9':'----.',
               '0':'-----', ', ':'--..--', '.':'.-.-.-',
               '?':'..--..', '/':'-..-.', '-':'-....-',
               '(':'-.--.', ')':'-.--.-', ' ': ' '}

This dictionary will allow you to write a function which steps through a string, converting each character to Morse code as it goes. Here's a function which does just that:

def display_message(msg):
  morse = "".join([MORSE_DICT[m] for m in msg.upper()])
  return morse

Now, you'll want to be able to call this function by clicking a button in your app's user interface, which means you need to make it callable from the client. To do this, add the anvil.server.callable decorator to your function, so that it looks like this:

@anvil.server.callable
def display_message(msg):
  morse = "".join([MORSE_DICT[m] for m in msg])
  return morse

Finally, you're going to want this server function to call the morse_message function on your Pico with the Morse code string it's generated. To do that, replace the return statement with another anvil.server.call:

@anvil.server.callable
def display_message(msg):
  morse = "".join([MORSE_DICT[m] for m in msg])
  anvil.server.call('morse_message', morse)

Building a user interface for your app

In order for your app to take user input to display as Morse on your Pico W, it will need a user interface. In the Anvil Editor, open up Form1 from your app's structure menu:

This will open the web page you built as part of the Getting Started guide. So far, it only has a single button. Make sure you're using the Design view for your form:

Drag a TextBox from the component toolbox on the right, and drop it above the button.

To edit its properties - like its placeholder text, and the name of the Python object that represents it in client code - you can use the Properties panel on the right, below the component toolbox.

In the above GIF, the Python name is being set to self.txt_message and the placeholder text is being set to 'Type your message here!'.

You'll see the changes appear on your Form as you alter the properties of your components.

You can do the same thing with the button that's already on the page. Click it once, and the Properties panel will change to show the properties of the button, so that you can edit them. Set the 'text' property to 'Send message'.

Now, double-click the button, and you'll be taken to the function that runs when the button is pressed. If you followed the Getting Started tutorial, this will already have some code in it. Change this function so that it calls the display_message function in your Server Module:

  def button_1_click(self, **event_args):
    """This method is called when the button is clicked"""
    anvil.server.call('display_message', self.txt_message.text)

(Your function's name might be different if you changed the Python name of the button component in the Properties panel.)

Putting it all together

Now, click the 'Run' button in the top right of the Editor. Type in 'SOS' to your text box, click the 'Send message' button in your app, and watch the LED on your Pico W flash the Morse code!

Adding user authentication

Now, you might not want just anyone having access to the ability to flash LEDs on your Pico. This section covers adding user authentication to your web app, so that only authorised users can enter text and have it sent as Morse code to your Pico.

Adding the Users Service

Click the '+' button in the lower left of the Editor to bring up the list of available services for your app, and select the Users service. This will bring up the configuration page for the Users Service in your app.

From here, you can choose the possible authentication methods for your users - such as email address and password, or single-sign-on with various providers like Google and Facebook. For now, leave the authentication method as 'Email + password'.

Under 'New user registration', check the box that allows new visitors to sign up for an account, and uncheck the box that allows their accounts to be used right away. If you like, you can require your users to confirm an email address before their accounts can be activated.

This configuration lets new users sign up for accounts, but requires you (as the app's owner) to manually enable accounts before they can be used.

Go to the code for 'Form1' by clicking on the 'Code' view button in the top left of the middle section of the Editor. Before, you've edited the code that runs when your button is pressed, but now you're going to add a line of code to your Form's __init__ method. This method runs whenever someone opens this Form - so, for example, when a user first visits your web app.

Change the __init__ method so that you require users to log in when the Form first opens:

  def __init__(self, **properties):
    # Set Form properties and Data Bindings.
    self.init_components(**properties)

    # Any code you write here will run when the form opens.
    anvil.users.login_with_form() # Add this line here

Now, click the 'Run' button in the top right of the Editor. You'll see a pop-up asking you to either log in or to sign up for a new account. Sign up for a new account using the email and password of your choice. If you've opted to require email confirmation, you'll need to do that before your account can be activated. Once you've done that, you should see a new row in the table at the bottom of the Users Service configuration page, like this:

You'll see that the 'enabled' checkbox is not yet checked, which means that you can't yet use this account to log in to your app. Check this box, and then click the 'Run' button again, and you will be able to use this account to log in. Then, you'll see the UI you previously built, and you can send messages to your Pico!

Finally, if you want an extra layer of security, you can change the code in the main.py file on your Pico W. Open it up, and add require_user=True to the decorator on your morse_message function, like this:

@anvil.pico.callable_async(require_user=True)
async def morse_message(msg):
    ...

This means that whenever a call is made to this function, Anvil will check that it came from an authenticated user in your web app. Save and close this file, then reboot your Pico.

Now, go to the public URL for your app that you chose while publishing your app in the Getting Started tutorial. You'll be prompted to log in, which you can now do with the account you created and enabled, and you'll be able to enter a text message and see the Morse code flash on your Pico!

And that's it! You've connected your Pico to a web app that takes user input and processes it, and uses the results to flash the LED on your Pico appropriately - and you've made sure that it's only available to the users you choose.

More about Anvil

Try Anvil - it's free, forever.

Pointers? In My Python? It's More Likely Than You Think - Part 3: Object Lifetimes and Garbage Collection

Eli Holderness — Wed, 24 Aug 2022 14:57:33 +0000

This is the third of a three-part series which covers various aspects of Python's memory management. It started life as a conference talk I gave in 2021, titled 'Pointers? In My Python?' and the most recent recording of it can be found here.

Check out Part 1 and Part 2 of the series - or read on for an discussion of object lifetimes, reference counting, and garbage collection in CPython!

How CPython can tell when you're done with an object, and what happens next

We ended Part 2 by asking the questions: once we've created an object x, how and why does its 'lifetime' end? In this article, we'll learn the answers by exploring how CPython frees objects from memory. CPython isn't the only implementation of Python - for example, there's Skulpt, which Anvil uses to run Python in the browser - but it's the one we'll focus on specifically for this article.

We ended Part 2 with an exploration of the weird and wonderful things that can happen when you override the __eq__ magic method in Python. Now, in Part 3, we're going to look at doing the same thing with a different magic method: __del__.

The `del` magic method

The __del__ magic method, also called the finaliser of an object, is a method that is called right before an object is about to be removed from memory. It doesn't actually do the work of removing the object from memory - we'll see how that happens later. Instead, this method is meant to be used to do any clean-up work that needs to happen before an object is removed - for example, closing any files that were opened by the object when it was created.

We're going to be using the following class as an example throughout this section:

class MyNamedClass:
  def __init__(self, name):
    self.name = name

  def __del__(self):
    print(f"Deleting {self.name}!")

This is just a class that'll let us know when one of its instances is about to be removed from memory - or, more specifically, when Python expects to immediately remove the class instance from memory (this won't always be true, as we'll see!).

In the above example, we've defined our class to take a name input when initialised, and when the finaliser is called, it'll let us know by printing the name of the instance in question. That way, we can get a bit of insight into which of these objects are being removed from memory, and when.

So, when will CPython decide to remove an object from memory? There are (as of CPython 3.10) two ways this happens: Reference Counting and Garbage Collection.

Reference counting in CPython

If we have a pointer to an object in Python, that's a reference to that object. For a given object a, CPython keeps track of how many other things point at a. If that counter reaches zero, it's safe to remove that object from memory, since nothing else is using it. Let's see an example:

>>> jane = MyNamedClass("Jane")
>>> del jane
Deleting Jane!

Here we create a new object (MyNamedClass("Jane")) and create a pointer that points at it (jane =). Then, when we del jane, we remove that reference, and the MyNamedClass instance now has a reference count of 0. So, CPython decides to remove it from memory - and, right before that happens, its __del__ method is called, which prints out the message we see above.

If we create multiple references to an object, we'll have to get rid of all of them in order for the object to be removed:

>>> bob = MyNamedClass("Bob")
>>> bob_two = bob # creating a new pointer to the same object
>>> del bob # this doesn't cause the object to be removed...
>>> del bob_two # ... but this does
Deleting Bob!

Of course, our instances of MyNamedClass could themselves contain pointers - after all, they're arbitrary Python objects, and we can add whatever attributes we like to them. Let's see an example:

>>> jane = MyNamedClass("Jane")
>>> bob = MyNamedClass("Bob")
>>> jane.friend = bob # now the "Jane" object contains a pointer to the "Bob" object...
>>> bob.friend = jane # ... and vice versa

What we've done in the above code snippet is set up some cyclic references. The object whose name is Jane contains a pointer to the one whose name is Bob, and vice versa. Where this gets interesting is when we do the following:

>>> del jane
>>> del bob

We've now remove the pointers that go from the namespace to the objects. Now, we can't access those MyNamedClass objects at all - but we didn't get the print message telling us they're about to be deleted. This is because there are still references to these objects, contained within each other, and therefore their reference counts are not 0.

What we've created here is a cyclic isolate; a structure where each object has at least one reference within the cycle, keeping it alive, but none of the objects in the cycle can be accessed from the namespace.

Below is a visual representation of what's going on when we create a cyclic isolate.

To begin, we create our two objects, each of which also has a name in the namespace.

Next, we connect our two objects by adding a pointer from each to the other.

Finally, we remove the pointers from the namespace by removing both of the original names for our objects. At this point, the two objects are inaccessible from the namespace, but each contains a pointer to the other so their reference counts are not zero.

So, clearly, reference counting on its own isn't sufficient for keeping the working memory of your runtime free of useless, irretrievable objects. This is where CPython's Garbage Collector comes in!

Collecting garbage in CPython

CPython's Garbage Collector (or GC for short) is Python's built-in way to get around the problem of cyclic isolates that we just encountered. By default, it's always running in the background, and it'll work its magic every now and then so you don't have to worry about cyclic isolates clogging up your memory.

The garbage collector is designed to find and remove cyclic isolates from CPython's working memory. It does this in the following way:

It detects cyclic isolates
It calls the finalisers (the __del__ methods) on each object in the cyclic isolate
It removes the pointers from each object (thus breaking the cycle) - only if the cycle is still isolated after step 2 (more on this later!)

After this process is complete, every object that was previously in the cycle will now have a reference count of 0, and therefore will be removed from memory.

Although it works automatically, we can actually import it as a module from the standard library. Let's do that, so we can take an explicit look at how it works!

>>> import gc

Detecting cyclic isolates

CPython's garbage collector keeps track of various objects that exist in memory - but not all of them. We can instantiate some objects and see whether the garbage collector cares about them:

>>> gc.is_tracked("a string")
False

>>> gc.is_tracked(["a", "list"])
True

If an object can contain pointers, that gives it the ability to form part of a cyclic isolate structure - and that's what the garbage detector exists to detect and dismantle. Such objects in Python are often called 'container objects'.

So, the garbage collector needs to know about any object that has the potential to exist as part of a cyclic isolate. Strings can't, so "a string" isn't tracked by the garbage collector. Lists (as we've seen) are able to contain pointers, and therefore ['a', 'list'] is tracked.

Any instance of a user-defined class will also be tracked by the garbage collector, as we can always set arbitrary attributes (pointers) on them.

>>> jane = MyNamedClass("Jane")
>>> gc.is_tracked(jane)
True

So, the garbage collector knows about all the objects that could potentially form a cyclic isolate. How does it know if one has formed? Well, it also knows about all the pointers in each of those objects, and where they point. We can see this in action:

>>> my_list = [“a”, “list”]
>>> gc.get_referents(my_list)
[‘list’, ‘a’]

The get_referents method (also called a traversal method) takes an object, and returns a list of the objects it contains pointers to (its referents). So, the list above contains pointers to each of its elements, which are both strings.

Let's take a look at the get_referents method in the context of a cycle of objects (not yet a cyclic isolate, though, since these objects can still be accessed from the namespace):

>>> jane = MyNamedClass("Jane")
>>> bob = MyNamedClass("Bob")
>>> jane.friend = bob
>>> bob.friend = jane
>>> gc.get_referents(bob)
[{'name': 'bob', 'friend': <__main__.MyNamedClass object at 0x7ff29a095d60>}, <class '__main__.MyNamedClass'>]

In this cycle, we can see that the object pointed to by bob contains pointers to the following: a dictionary of its attributes, containing bob's name (bob) and its friend (the MyNamedClass instance also pointed at by jane). The bob object also has a pointer to the class object itself, since bob.__class__ will return that class object.

When the garbage collector runs, it checks whether every object it knows about (that is, anything that returns True when you call gc.is_tracked on it) is reachable from the namespace. It does this by following all the pointers from the namespace, and pointers within the objects that those point to, as so on, until it builds up an entire view of everything that's accessible from code.

If, after doing this, the GC finds that there exist objects which aren't reachable from the namespace, then it can clear those objects up.

Remember, any objects that are still in memory must have a non-zero reference count, or else they'd have been removed due to reference counting. For objects to be unreachable and yet still have a non-zero reference count, they have to be part of a cyclic isolate, which is why we care so much about the possibility of these occurring.

Let's return to our cycle of friends, jane and bob, and turn that cycle into a cyclic isolate by removing the pointers from the namespace:

>>> del jane
>>> del bob

Now, we've got ourselves into the exact situation that the garbage collector exists to fix. We can trigger manual garbage collection by calling gc.collect():

>>> gc.collect()
Deleting Bob!
Deleting Jane!
4

By default, the garbage collector will perform this action automatically every so often (as more and more objects are created and destroyed within the CPython runtime).

The output that we see in the code snippet above contains the print statements from our MyNamedClass's __del__ method, and at the end there's a number - in this case, 4. This number is output from the garbage collector itself, and it tells us how many objects were removed.

You might think that only 2 objects (our two MyNamedClass instances) were removed, but each of them also pointed to a string object (their name). Once those two MyNamedClass instances are removed, the reference count for each of those name strings also falls to zero, so they're removed too, bringing the total to 4 objects.

Finalisers behaving badly

Earlier, we mentioned that the garbage collector works in a 3-step process: detecting cyclic isolates, calling the finalisers on each object in the cycle, then breaking the cycle by removing the pointers between the objects... if the cycle still remains isolated at this point. Now, the only way that the cycle could go from being isolated to not-isolated between the first and third step is if the finalisers do something to make that happen.

Let's define a class that does just that:

class MyBadClass:
  def __init__(self, name):
    self.name = name

  def __del__(self):
    global person # create an externally accessible pointer...
    person = self # ... and point it at the object about to be removed
    print(f“deleting {self.name}!”)

In this class's finaliser, a global variable is created. That means that even if an instance of MyBadClass becomes inaccessible from the namespace (as part of a cyclic isolate, for example), it can still 'reach out' into the namespace, create a pointer there, and point that pointer at itself - thus de-isolating itself.

>>> jane = MyBadClass("Jane")
>>> bob = MyBadClass("Bob")
>>> jane.friend = bob
>>> bob.friend = jane
>>> del jane
>>> del bob

To see this in action, we set up the cyclic isolate structure, as we've done before with other (more well-behaved) classes. Then, we trigger garbage collection:

>>> gc.collect()
Deleting Bob!
Deleting Jane!
0

We see the print statements from each instance's __del__ method, but after that, the garbage collector prints us out a 0. That means that no objects were removed from memory - and that's because, after the garbage collector caused the finalisers to be called, it checked to make sure that the cycle was still isolated.

If the cycle were still isolated, then the garbage collector could safely remove all the pointers linking up the objects, reducing their reference counter to 0. But, in this case, the cycle was no longer isolated, and so the garbage collector doesn't break the links between the objects in it.

So, if we got rid of the jane and bob pointers, how can the cycle still be accessed from the namespace? The answer is that global person variable that was created in the finaliser. Let's take a look at it:

>>> person
<__main__.MyNamedClass object at 0x7ff29a095d60>

>>> person.name
'Jane'

>>> person.friend.name
'Bob'

We can see that the object pointed to by person is the same one that had previously been pointed to by jane. This make sense if you look at the above output from calling gc.collect(); the print statement that appeared last was the one for the Jane object, and therefore that was the object that set person = self most recently.

In other words, the two objects have had their original pointers jane and bob removed - but when their finalisers are called, a new external pointer from the namespace is created, meaning that the cycle is no longer isolated and shouldn't be removed by the GC.

Doing this sort of thing can create strange results, because it means you can access objects whose finalisers have already run -- and that probably means they've cleaned themselves up in a way that means you shouldn't be interacting with them again. For example, their finalisers may have closed a file that other methods on the object will assume to still be open. once again: overriding magic methods is serious business!

So, does MyBadClass break garbage collection entirely? The answer is no, and that's because of a very important property of finalisers: they can only be called once per object. After bob's __del__ method has been called once (when it was triggered by the call to gc.collect()), it's done, and can never be executed again. That means we can do the following:

>>> del person
>>> gc.collect()
4

We don't see the "Deleting Jane" and "Deleting Bob!" messages, because those are printed by the objects' finalisers - and those have already been called once, and can't be called again. But, since the person pointer has been removed, the cycle is isolated again; and, because that person pointer won't be recreated by a finaliser, the garbage collector can safely go ahead and remove the pointers linking our two MyBadClass instances.

Then the garbage collector continues its work, printing out a 4 to let us know that those two objects and their name attributes have been removed from memory - and all is well in the world (of our CPython interpreter, at least!) again!

So what have we learned?

Let's recap! These three articles have been a whistle-stop tour of how Python handles objects in memory. We've looked at how pointers work, why pointer aliasing happens, and whether you'll need to use =, copy or deepcopy. We've also seen what object IDs are that the is comparator uses them, and how we can override the __eq__ magic method to define our own equality conditions to make == do whatever we want. Finally, we've covered object lifetimes, the __del__ magic method, and how CPython frees objects from memory when they're not needed any more using reference counting and garbage collection.

Now that you know all of these things, you can go forth and write better Python code!

See this article as a talk

Various recorded versions of this talk are available at the following links:

More about Anvil

Try Anvil - it's free, forever.

Get Started with the Pico W

Eli Holderness — Fri, 22 Jul 2022 14:58:24 +0000

Build your first internet-connected device

This guide will show you how to take your brand new Pico W, connect it securely to the internet, and control it from from a web interface -- all in Python.

We're going to:

Set up your Pico W to use Anvil, by installing the Anvil firmware
Build your web app with the Anvil
Connect your Pico W to your web app
Update your web app to call your Pico W from the web
Publish your app on the web

Let's get started!

Setting up your Pico W

To get your Pico W ready to use, you'll need to install the Anvil firmware. You'll only need to do this once.

First, download the UF2 firmware file from here.

To install it onto your Pico W, you'll need to connect it to your computer via USB in firmware installation mode, which allows you to install new firmware from files on your computer. To connect your Pico W in firmware installation mode, you'll need to hold down the BOOTSEL button as you plug it into your computer.

This is where the BOOTSEL button on your Pico W is.

Once you've connected your Pico to your computer with the BOOTSEL button held down, it will appear as a drive on your computer:

This is how your Pico W will appear when it's in firmware installation mode.

To install the Anvil firmware, copy the UF2 file you downloaded across to your Pico W. Once it's finished copying and processing, your Pico will automatically reboot and reappear again as a drive on your computer (perhaps with a different volume name). If you check what files are now inside, you'll see a boot.py and a main.py.

If the USB drive doesn't appear, check out the Troubleshooting section of the reference docs for help.

This is how your Pico W will appear after you've installed the Anvil firmware.

That's it! You've now installed (or "flashed") the Anvil firmware onto your Pico W, and you're ready to start building.

Building your Anvil app

Next, you're going to build the web interface that's going to drive your Pico W. When you're done, you'll be able to click a button in this app and see lights flash on your Pico W!

If you've tangled with web development before, this might sound daunting -- but don't worry. You're going to do it all in Python, with a drag-and-drop UI designer and an online editor at anvil.works.

For now, all you need to do is create a new Anvil app and get an Uplink key to link it to your Pico.

Creating an Anvil app

Head to the Anvil Editor to create a new app (you'll need an Anvil account for this - it's free!). On the landing page of the Editor, click 'Create a new app', then 'Blank app'.

From the dialog that appears, choose 'Material Design'.

What you'll now see before you is a new, blank Anvil app. For now, all we need to do is add the Uplink, which is how your app will communicate with your Pico. To do this, click the '+' icon in the lower left.

Choose the Uplink from the list:

From the dialog box that appears, click 'Enable server Uplink'.

That will generate an authentication key that you can copy:

Copy the Uplink key from that dialog box - you'll need it in a minute. (You can always bring this dialog back by clicking the 'Uplink' button in the top right of the Editor.)

OK -- you now have a web app. It's time to connect your Pico to it!

Connecting your Pico W to your web app

Connect your Pico W to your computer via USB (don't hold down the BOOTSEL button this time!). As you've seen before, it will appear as a new drive, containing two .py files: boot.py and main.py.

This is how your Pico W will look as a drive on your computer once the Anvil firmware has been installed.

It's now time to edit those files to configure your app.

Adding your WiFi network details into `boot.py`

To let your Pico connect to the internet when it boots up, you'll need to give it your network details. Open up boot.py, and at the very top you'll find a section where you can add these details:

#############################################
# Provide your Wifi connection details here #
#############################################

WIFI_SSID = "<put your network name here>"
WIFI_PASSWORD = "<put your wifi password here>"

#############################################

Save and close that file. Now, when your Pico W boots, it'll automatically connect to the network you provided.

Adding your Uplink key into `main.py`

In order for your Pico W to know about the Anvil app you just created, you'll need to give it your Uplink key. Open up main.py: right underneath the import statements you'll see the section where you can add your key:

# Replace this string with your Uplink key!
UPLINK_KEY = "<put your Uplink key here>"

Edit the code to use your app's real Uplink key. Now, when your Pico has booted and connected to the internet, it will be able to listen for input from your Anvil app.

Understanding the `main.py` file

Your Uplink key isn't the only thing in main.py. This is also where you can define functions that your Pico can run. To start with, there's a pico_fn function, and all it does is toggle the LED on your Pico back and forth for a little bit.

@anvil.pico.callable_async
async def pico_fn(n):
    # Output will go to the Pico W serial port
    print(f"Called local function with argument: {n}")

    # Blink the LED and then double the argument and return it.
    for i in range(10):
        led.toggle()
        await a.sleep_ms(50)
    return n * 2

The anvil.pico.callable_async decorator lets your Anvil app know that this function is available to call from the web.

Now, you're all set! Save your code, and reboot your Pico W. You'll see the LED flash slowly until the WiFi connects, then quickly while the connection to Anvil is established.

You might want to connect to the USB serial device to see the Pico's output - see section 3.3 of the Pico W documentation for more information.

When connecting to the serial port of the Pico W, the running Anvil firmware means that you sometimes won't see the MicroPython prompt automatically. Just hit Ctrl-C to stop whatever it's doing (such as failing to connect to the Wifi before you've added your credentials to boot.py), and the prompt should appear.

Updating your app to call your Pico W

Open up your Anvil app in the Anvil Editor. From the component toolbox on the right, drag and drop a Button component onto your page. For now, you don't need to worry about changing any of its properties.

Now, double click that button in the Editor, and you'll be taken to the new Python function that runs when that button is clicked. All you need to do is make that function call the function that's in your main.py file on your Pico, and to do that, you use anvil.server.call() like this:

  def button_1_click(self, **event_args):
    """This method is called when the button is clicked"""
    anvil.server.call('pico_fn', 18) # Choose any number you like!

Now, to see this in action, click the 'Run' button in the top right of the Editor. This will run your app, and if you now click the button, you'll see the LED on your Pico W flashing.

You've successfully called a function on your Pico from your web app!

Publishing your app on the web

The very last step is make your app available on the public web. To do this, click the 'Publish' button in the top right of the Editor.

Click 'Publish this app' in the next dialog box:

That's it -- your app is online!

Anvil automatically generates a URL for your app, but you can choose a different one in the dialog box with the 'Change URL' button:

Choose a snappy URL for your app - or let Anvil generate one for you.

Once you've done this, you can open that URL from anywhere in the world -- your computer, your phone, Australia. Click the button, and make the LED on your Pico flash!

Go even further

Now that you've connected your Pico W securely to the web with Anvil, you can build whatever you want.

As a first project, let's make your Pico's LED flash Morse code messages entered from a web page! Here's a step-by-step tutorial:

Build a Morse Code Generator

Or, if you're ready to dive into your own projects, you can read about some of the more advanced things you can do with Anvil on your Pico W, or consult our reference documentation.

More about Anvil

Try Anvil - it's free, forever.

Pointers? In My Python? It's More Likely Than You Think - Part 2: Equality

Eli Holderness — Thu, 30 Jun 2022 13:00:15 +0000

This is the second of a three-part series which covers various aspects of Python's memory management. It started life as a conference talk I gave in 2021, titled 'Pointers? In My Python?' and the most recent recording of it can be found here.

Check out Part 1 of the series, or read on for an discussion of Object IDs in Python!

Object IDs, and why they matter

We ended Part 1 with the following question: how do we know when two Python objects are really the same object in memory? If we do b = deepcopy(a), how can we know for sure that it didn't just create a new pointer instead of a whole new object? The answer is Object IDs.

Python has a built-in id function with the following properties:

id(x) is an integer
id(x) != id(y) exactly when x and y point at different objects in memory
id(x) is constant for the lifetime of x - that is, as long as x remains in memory

There are many implementations of Python, and while the above three things must be true of the id function in each of them, they don't all do it in the same way under the hood. Some implementations, such as CPython (the python interpreter written in C), use the object's memory address as its id - but don't assume that all implementations will!

For example, Skulpt is the Python-to-JavaScript compiler which Anvil uses to run Python client code in the browser so you can develop for the web without having to write JavaScript; Skulpt's implementation of id generates and caches a random number for every object in memory.

For the rest of this article, we'll be using examples generated using CPython, which equates an object's id with its address in memory.

So, let's look at what happens when we check an object's id!

>>> a = ["a", "list"]
>>> id(a)
139865338256192

>>> b = a
>>> id(a), id(b)
(139865338256192, 139865338256192)

Here we've defined a list a, and created a new pointer to it by setting b = a. When we check their ids, we can see that they're the same - a and b point to the same thing.

>>> c = a.copy()
>>> id(a), id(c)
(139865338256192, 139865337919872)

Trying the same thing with c = a.copy() shows that this creates a new list object; a and c have different ids.

However, that isn't the only notion of 'sameness' that Python provides. Consider our familiar example, with a pointing to a list object list, b another pointer to that object, and c a pointer to a copy of that object:

>>> a = ["my", "list"]
>>> b = a
>>> c = a.copy()

With this setup, we can do the following comparisons:

>>> a == b
True
>>> a is b
True

>>> a == c
True
>>> a is c
False

Once again: what is going on here? How can two things be the same and not the same? The answer is that is and == are designed to serve two different purposes. is is for when you want to know if two pointers are pointing at the exact same object in memory; == is for when you want to know if two objects should be considered to be equal.

`is` uses `id(x)`

Saying a is b is directly equivalent to saying id(a) == id(b). When you call is on two objects, Python takes their ids and directly compares them.

`==` uses `eq`

When you write a == b, you're actually calling a magic method, also known as a dunder method (named for the double-underscore on each side). You might be familiar with some magic methods already, such as:

__init__, called when an instances of a Python class is initialised
__str__, called when you use the str built-in - e.g. str(some_object)
__repr__, similar to __str__ but also called in other circumstances such as error messages

Magic methods are simply methods on a Python class, and the double underscores indicate that they interact with built-in Python methods. For example, overwriting the __str__ method on a Python class would change how the str built-in behaved if you called it on an instance of that modified class.

When it comes to == and __eq__, it's easiest to understand with some examples. Let's dive in!

class MyClass:
  def __eq__(self, other):
    return self is other

Here we've defined a custom class with its own __eq__ method. Every __eq__ method takes two arguments including self - because whenever it's called, it'll be comparing two objects, including the instance of the class in question. In the above example, we've just set the method to fall through to the is definition of equality (comparing the ids of each object). As it happens, this is actually the default behaviour for any user-defined class in Python.

So, what happens if we define some non-default behaviour?

class MyAlwaysTrueClass:
  def __init__(self, name):
    self.name = name

  def __eq__(self, other):
    return True

Here we've defined a class which takes a name argument (so we can keep track of our instances!) and has an __eq__ method which indiscriminately returns True. This gives us the following behaviour:

>>> jane = MyAlwaysTrueClass("Jane")
>>> bob = MyAlwaysTrueClass("Bob")
>>> jane.name == bob.name
False

>>> jane == bob
True

Because we overrode the __eq__ method to always return True, that means that all instances of this class will be considered equal under the == comparator - even when their names have different values!

Conversely, we can also do the following:

class MyAlwaysFalseClass:
  def __init__(self, name):
    self.name = name

  def __eq__(self, other):
    return False

You might think this is more sensible, but consider:

>>> a = MyAlwaysFalseClass("name")
>>> a == a
False

Moreover, because the behaviour of __eq__ is dependent on which object is self and which is other, we can get the following:

>>> jane = MyAlwaysTrueClass("Jane")
>>> bob = MyAlwaysFalseClass("Bob")
>>> jane == bob
True

>>> bob == jane
False

In summary: magic methods are a fun way to make Python do things that seem very strange.

Earlier, we mentioned that id(x) is constant and unique 'for the lifetime of x', which is equivalent to saying 'as long as x remains in memory'. That raises the question: once we've created an object x, how does its 'lifetime' end? Hold on 'til Part 3 of this series, where we'll get the answer!

More about Anvil

Try Anvil - it's free forever!

Building A Python Code Completer at PyCon US 2022

Eli Holderness — Thu, 26 May 2022 14:23:47 +0000

Almost all of Anvil went to PyCon US in Salt Lake City this past April, and we had a blast. To round off the trip, we were able to watch Meredydd give his first in-person talk at a US PyCon - and it was awesome. Even better, the recordings are now out, so you can see it for yourselves!

In his talk, Meredydd teaches us how Python parses and compiles code, what an AST is, and how we can use this knowledge to work out what a programmer might type next. And, to prove it’s not that complicated, he builds a little code completer, live on stage, in about five minutes.

More about Anvil

If you’re new here, welcome! Anvil is a platform for building full-stack web apps with nothing but Python. No need to wrestle with JS, HTML, CSS, Python, SQL and all their frameworks – just build it all in Python.

Try Anvil - it's free, forever.

Pointers? In My Python? It's More Likely Than You Think

Eli Holderness — Mon, 23 May 2022 14:56:22 +0000

This is the first of a three-part series, covering various aspects of Python's memory management; it started life as a conference talk I gave in 2021, titled 'Pointers? In My Python?', the most recent recording of which can be found here.

A tour of Python's memory magic

Python is a wonderful thing that takes all the complication of memory management away from us. We don’t have to worry about pre-allocating memory for our objects, or remember to free it once we’re done. So, given that we’re not doing it manually, how do these things happen? Do we have to care? Well, sometimes. Maybe.

For example, ever wondered about the difference between is and ==, or why you might need to use deepcopy? Maybe you’ve been stumped by a variable changing when you didn’t expect it to, or an interview question about object lifetimes. Or, perhaps, you just really want to see some tuples behaving badly. This three-part series answers all these questions and more, covering the following:

Part 1: what a pointer is, and where you'll see them in Python
Part 2: what the id of a Python object is, and why it matters
Part 3: how CPython can tell when you're done using an object in memory, and what it does next

Let's dive in with Part 1!

What pointers are, and where you'll find them

Firstly, we need to understand the concept of a namespace. A namespace in Python is the list of all the variables, keywords and functions that are in scope at any given point - that is, things you can write that the Python interpreter will understand. For example, all the built-in functions like print() and str(), and keywords like None and True are always in every namespace.

When you create a new variable, then that variable's name is added to the namespace of whichever scope it's in. So, for example, writing the following will add the name my_string to the global namespace:

>>> my_string = "Hello World!"

For the purposes of this series of articles, we don't need to worry about scopes; we can assume all our examples take place in the global namespace.

Pointers can be thought of as names - that is, entries in Python's namespace - that correspond to objects in Python's memory. In the above example, the pointer is my_string, and the object in memory is the string with value "Hello World!".

By using a pointer in namespace, you can access and manipulate the object in memory. And, just as a person might have multiple names, multiple pointers might point to the same object.

A note about terminology: the 'pointers' referred to in this article are not directly equivalent to pointers in C or C++ (in fact, they're more similar to references in C++). For those brave souls who code in C, an excellent breakdown of the minutiae can be found here.

As an example, let's consider a list object with the name my_list and two arbitrary elements.

my_list = ['string', 42]

That name my_list then points to the list object. That list object then contains pointers to the two objects which are the elements of that list. So, when you create a list, it will automatically contain pointers if it has any elements. For that reason, we'll be using a lot of lists as examples throughout this article.

Pointer aliasing

One Python behaviour that often trips up a lot of beginners is something called pointer aliasing, which is when two pointers refer to the same object in memory. Let's look at an example: a list containing some strings.

>>> a = ["string", 42"]
>>> a
["string", 42]

Here, we've defined our list a and got our interpreter to print it back out for us, just to check that it is as we expect. Next, we (naively) try to make a copy, and make some changes to it, namely changing "string" to "some words":

>>> b = a
>>> b[0] = "some words"
>>> b
["some words", 42]

Great! Except, it turns out we've also changed our original list a:

>>> a
["some words", 42]

The common misconception here is that a is the list object, when it's actually just pointing at it, and might not be the only that that points at it. What's happened above is that, in the line where we set b = a, we didn't actually make a new list object. We just created a new pointer, b, and made it point to the same underlying list object that a already pointed to.

So, when we change b, we're changing a too. But what if we did want to make a new list object, and be able to make changes to it without affecting the original? Well, there's a list method for that:

>>> c = a.copy()
>>> c[0] = "hello!"
>>> c
["hello!", 42]

>>> a
["some words", 42]

We can use the copy method on our original list object, and this does create a new list object. That new list object will also contain new pointers - but, those pointers will then point to the same underlying elements of the original list.

Using copy (whether as a list method or as a function from within the copy module) creates a new object, and populates it with new pointers to the existing elements.

The outer list object - the thing that also has access to list methods, and which contains pointers to its contents, is different - but with copy, the elements of each list will still be the same objects in memory. So, what if those elements are themselves lists?

Let's define a new list:

>>> a = [["alex", "beth"]]
>>> a
[["alex", "beth"]]

Here's a visual representation:

Now, let's do as we did before, and make a new list object using the copy method. This time though, we'll append something to the first element of b, not b itself.

>>> b = a.copy()
>>> b[0].append("charlie")
>>> b[0]
["alex", "beth", "charlie"]

So far, so good, right? Except....

>>> a[0]
["alex", "beth", "charlie"]

... we managed to alter the contents of a, even though we used the copy method. This is because, as stated above, the pointers in b still point at the same contents as the original list - so, we get the same pointer aliasing behaviour as we saw in the very first example, just one layer deeper. This kind of copy (only creating new objects one level deep, and pointer aliasing the rest) is called a 'shallow copy'.

So, if we want to make a true, 'deep' copy - that is, to make not only a new list object, but new versions of all its contents - how do we do that? The answer is deepcopy, a function within the copy module of the standard library.

>>> from copy import deepcopy
>>> c = deepcopy(a)
>>> c[0].append("dan")
>>> c[0]
["alex", "beth", "charlie", "dan"]

>>> a[0]
["alex", "beth", "charlie"]

What deepcopy does is recursively create new versions of every object it encounters - so, when it's called on our list a, it'll create a new list, and when it sees that the elements of a also contain pointers themselves, it'll make new copies of the things those pointers point to as well. (Try saying that three times fast with a mouthful of spaghetti.)

Using deepcopy creates a whole new inner list, complete with new contents. So, when we mutate the inner list of c, it's not touching the original inner list that a points at - because deepcopy made a new, separate copy of that list when it created c.

In this particular scenario, we're using strings as the contents of our inner lists. That means that technically the alex string pointed to by a[0][0] is also the same object in memory as the one pointed to by c[0][0], because Python has some memory optimisations that prevent it from creating the same immutable object twice if it doesn't need to. If we'd used - for example - a user-defined class object instead of strings, then deepcopy would have caused Python to make new instances of those objects too.

If we had an object with more layers of pointer nesting, such as a list containing a list containing a list, then deepcopy would make an entirely new copy of that entire object and all its contents, all the way down, with no pointer aliasing.

Immutable Objects (or: Tuples Behaving Badly)

So far, we've been looking at lists, which are mutable objects. What happens if we look at something immutable, like a tuple?

If we say that a tuple a is immutable, what we mean by that is that when a is created, all its elements - a[0], a[1], and so on - are fixed. If its elements are immutable, like strings or integers, it's fairly simple to understand what this means.

>>> a = (42, 'beeblebrox')
>>> a[0] = 63
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

If we try to change the value of a[0], we get an error. But, what if a[0] is a pointer to something mutable, like a list?

>>> a = ([1, 2, 3], "hello")
>>> a[0].append(4)
>>> a
([1, 2, 3, 4], "hello")

As it turns out, we can mutate the elements of a in-place with no problem! This is because we're not changing the value of a[0] itself - it's just a pointer. What we're changing is the value of the object that a[0] points to. If we gave a[0] its own name - a pointer alias - this would become a bit clearer:

>>> my_list = [1, 2, 3]
>>> a = (my_list, "hello")
>>> my_list.append(4)
>>> a
([1, 2, 3, 4], "hello")

However, the append method isn't the only way to add to a list!

The `+=` operator

We can do the following:

>>> my_list = [1, 2, 3, 4]
>>> my_list += [5, ]
>>> my_list
[1, 2, 3, 4, 5]

Here, we're using the += operator, which does the following:

First, it creates the desired object. For mutable objects, like a list, it does this by mutating the object in-place. For immutable objects, like strings, it creates an entirely new object. This is the '+' part of the operation.
Secondly, it reassigns the pointer it was given (in the above example, my_list) to point at the desired object.

If += is called on a mutable object, then Step 2 is pretty redundant - after all, the pointer is already pointing at the desired object. But when it's called on something immutable - like a string - it does need to change where the pointer points. For example:

>>> my_string = 'Hello'
>>> my_string += ', World!'
>>> my_string
'Hello, World!'

Strings aren't mutable, so in Step 1, += creates an entirely new string 'Hello, World!' and changes the my_string pointer to point at it.

Here's a visual representation:

So what if we try the += operator with the first element of our tuple a? Spoiler alert: something silly is about to happen!

>>> a = ([1, 2, 3, 4], "hello")
>>> a[0] += [5, ]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

>>> a
([1, 2, 3, 4, 5], "hello")

What on earth is going on there? We get an error when we try to use the += with a tuple element, but the operation seems to have gone through anyway; the value of a[0] has changed, at least.

Step 2 is where we fall over: in this case, we can't assign directly to a[0], since it's an element of a tuple. In the my_list example, however, there was no problem at all, since we can set my_list to point at whatever we like.

But, in step 1, the list object that a[0] points at was mutated in-place, which is the change we wanted to happen. Then, in Step 2, += assigns to the pointer it's called on - a[0] - and we can't assign to that! So, we get both the change and the error.

So, what have we learned? We've covered namespaces, what pointers are, and where you'll see them in code, along with some examples of how immutability and pointers can interact in confusing ways. But we're only scratching the surface - stay tuned for Part 2, where we'll learn about Object IDs and why they matter, how Python knows when two objects are really the same, and the difference between is and ==.

More about Anvil

Try Anvil - it's free, forever.

OAuth and OIDC: What You Need To Know

Eli Holderness — Fri, 08 Apr 2022 14:11:35 +0000

OIDC under the microscope

In a previous blog post we covered SSO, what it is, and the two main technologies that are used to implement it. In the next post, we covered SAML, the first of these technologies. This time, we’re taking a deep technical dive into the other: OpenID Connect - often abbreviated to OIDC - and OAuth, the authorization protocols that support it.

This blog post covers the following:

What OAuth is, and what problems it was built to solve
Why it can't be used for authentication
What OpenID Connect is, and how it uses OAuth
Importantly, is it secure?

Let's jump in!

What is OAuth?

This section gives a fairly high-level primer on OAuth, intended to illustrate what it is in the context of OIDC. In and of itself, OAuth is a deep topic with lots of interesting security aspects; for the sake of brevity, we won't be covering all of them.

Wikipedia defines OAuth (short for Open Authorization) as 'an open standard for access delegation'. In this context, 'access delegation' means allowing one entity access to something (for example, information) controlled by another entity. The act of allowing this access is delegation, hence 'access delegation'.

... so what does it do?

Let's take a real-world example: I have a GMail account, and I want to use a service that'll go through all the automated emails I received in the last 90 days and send responses asking each of the senders to delete the data they have on file about me. That means that the service will need to be able to read my emails, and send emails from my account. However, I don't want to just give the service the credentials I use to log in to my Google account; that would also give it access to my Drive, any location data history the Google Maps has collected, and so on.

Instead, the service should only be able to perform the actions I want it to, and that's where OAuth comes in. OAuth provides a mechanism for allowing that service to request access specifically to only the actions it needs, without compromising the security of other information or restricted actions managed by my Google account.

In OAuth world, there are a few different entities:

The user is called the Resource Owner: in the above example, this is me! I own the resources (the ability to read and send email) that I want to be able to share with the client service.
The client service is called the Relying Party: in the above example, this is the service that'll automatically read and send emails for me, and it relies on an OAuth flow in order to get access to those actions.
The Identity Provider is also called the Authorisation Server: This is where I can go to prove ownership of my Google account, and authorise the client to use my resources.
The Resource Server is where those resources to which I have delegated access live. In the above example, this is a GMail server. In a lot of examples, this will be managed by the same organisation as the Authorisation Sever, but that isn't strictly necessary.

How does it work?

The big concept that drives OAuth is the idea of tokens. Tokens are objects that the Authorisation Server can issue for a client to use, and in OAuth there are two different kinds:

Access Tokens are used by the client to access a given resources from the Resource Server. The Resources Server needs to be able to parse the information from within the token, and give exactly the access requested (and no more!). Access Tokens are also sometimes called bearer tokens, because any entity which gets hold of them ('bears' them) will then have access to those resources. For this reason, they should have an expiration date built-in, after which point the Resource Server will deny access.
Refresh Tokens are used by the client to get a new Access Token if the one they currently have has expired. These tokens are longer-lived, and are typically stored by the client application in a secure server.

There are a few different important qualities that tokens need to have:

They should be opaque to the client application. The email service in the above example doesn't need to know what's inside the tokens it receives; all it needs to do is send them on to the relevant servers.
When the Relying Party makes a request for a token, that request will include a scope (or multiple scopes) for the token that determines exactly what resources it grants access to. OAuth is a standard and not a framework, so it doesn't set out any scopes explicitly, instead leaving them up to the internal implementation of the Authorisation and Resource Servers. In the email service scenario, the scopes involved would require allowing access to reading and sending emails, but not (for example) access to my Google Drive.

An OAuth Authorisation Server will have a few different endpoints which are used at various stages in the process. In the following flow, we'll see a couple:

/authorize, which lets the user grant access - for example, https://my-oauth-server.com/authorize
/token, which is where the Relying Party sends its requests for tokens - for example, https://my-oauth-server.com/token

The Resource Server will also have a variety of endpoints, depending on what kind of resources it serves; it's up to the Relying Party to make sure they access the correct one.

So, authorising a client to use some of your resources that are hosted on a Resource Server goes like this:

You, the Resource Owner, visit the Relying Party that wants access to your resources.
- In the above example, this would be visiting the website of the service that'll automatically ask people to delete my data.
The Relying Party sends you to the Authorisation Server's /authorize endpoint with a request for an authorisation code.
- This is where you'll be redirected in your browser to a Google screen, asking you if you want to allow the email service appropriate access.
You authenticate with the Authorisation Server and grant resource access. Your role as the Resource Owner is now done.
The Authorisation Server sends you back to the Relying Party along with an authorisation code.
- In the above example, this is you being redirected back to the website for the email service.
The Relying Party then uses that code to request an Access Token with appropriate scopes from the Authorisation Server's /token endpoint.
The Authorisation Server then mints a new Access Token and (optionally) a Refresh Token.
The Authorisation Server sends those tokens back to the Relying Party.
The Relying Party sends the Access Token to a relevant endpoint at the Resource Server in order to ask for access.
- This endpoint is where the email service asks to download the last 90 days' worth of emails to my inbox.
The Resource Server validates the token and grants access.
- This is the point at which access to my emails has now been successfully delegated to the automatic email service.

Then, going forwards, as long as the client application has a valid Access Token (either from this process or by requesting a new one using its Refresh Token), it doesn't need you as the Resource Owner to grant access again. If the Refresh Token expires or is revoked, then this flow will need to happen again.

And that's how OAuth allows you to delegate access to your resources!

Why do we need something else for authentication?

Reading the above, you might think that OAuth is sufficient for authentication purposes. After all, if you want to use an OAuth-provided identity to log in with a third party service, OAuth already exists to facilitate permissions sharing between those two services, right? Well, that's just the thing: an OAuth token creates a relationship between those two services, letting one use the other's resources. You, as the user, are really only there at token creation time in order to agree that that relationship is OK; after that, OAuth doesn't care whether or not you're authenticated.

OAuth isn't about identity, it's about resources, and the authorisation to use those resources. These concepts are closely connected, but they're not the same, and getting them mixed up can lead to gaps in security. It's absolutely possible to do authentication with OAuth, and it's people did before OIDC - but you have to be very careful when implementing it.

For example, it might be tempting to assume that a valid Access Token constitutes proof of identity. After all, the user had to authenticate with the OAuth identity provider in order for that Access Token to exist. Unfortunately, there are multiple issues with this approach. One problem is that user authentication isn't the only time that Access Tokens can be created; any client with a Refresh Token can create one. This is in fact a fundamental design requirement of OAuth, since it exists to ease the sharing of resources between services, and shouldn't require user presence at all stages!

Another issue is that, since Access Tokens are opaque to the client by design, the third-party service that receives the Access Token as proof of identity won't be able to get any information from it. In fact, the service won't even be able to inspect the token to verify that the user is who they say they are! This is because the client isn't actually who the Access Token is for - it's meant to be parsed by the OAuth Resource Server, not the third-party service itself.

OAuth isn't unusable for authentication - but we can do better

In the past, we worked around all this by having the client follow the above OAuth Flow, after which it would use the resulting Access Token to hit an endpoint on the Authorisation Server. This endpoint, often located at /me, would then allow the Authorisation Server to parse that Access Token and tell the Relying Party on which user's behalf it was issued. But, this is fairly roundabout, and easy to get wrong; for example, when building a Relying Party you'd need to make sure that you only used Access Tokens received directly from the Authorisation Server in order to mitigate the risk of a malicious actor using an ill-gotten token to falsely authenticate.

What if we had something designed for the purpose? We'd need a token that's only created at authentication time, and is designed to be parsed by the client.

That's what OpenID Connect does!

What is OpenID Connect?

The key thing that OIDC provides is the idea of an ID Token - a token designed to be used as proof of OAuth identity, for consumption by the client service. ID Tokens, unlike Access and Refresh tokens, have a specified format: they're constructed as JSON Web Tokens (or JWTs). JWTs are JSON payloads which are then cryptographically signed by the issuer. This means that they're protected from tampering, but they can still be parsed by the recipient.

The flow for a user authenticating with a Relying Party using their OAuth Identity then looks like this:

You, the Resource Owner, visit the Relying Party with which you want to authenticate.
The Relying Party sends you to the Authorisation Server's /authorize endpoint, along with a request for one or more tokens.
- At this point, the Relying Party can request either an ID Token on its own, or an ID Token and an Access Token.
You authenticate yourself with the Authorisation Server.
The Authorisation Server then mints the tokens that the Relying Party requested, and sends them back to the Relying Party.
The Relying Party can now decrypt the ID Token, and use it to verify your identity and authenticate you.
- If the Relying Party also requested an Access Token, it can then use this later to get more user information about you from the Authorisation Server by using it to hit a /userinfo endpoint.

Note that the Resource Server isn't involved at all. The only 'resource' you're trying to share is proof of your identity, and that's solely the domain of the Authorisation Server.

In reality, the ID Token is quite different to the Access and Refresh Token types. It isn't meant to be used by the Resource or Authorisation Servers; it's just a record of the authentication event that occurred when the user logged in to the Authorisation Server. But that's all that the Relying Party needs in order to know who you are.

If the Relying Party also wants to know extra information about you, then it needs to request an Access Token which has the relevant scopes - for example, email and address - and it can do this at the same time, as mentioned in the flow above. (It might seem like one could just use the /userinfo endpoint to use OAuth itself for authentication, but that endpoint is in fact something that's defined by the OIDC spec, and isn't required for OAuth.)

OIDC and security

So, how can this all go wrong? Well, there are a few different ways, including the following:

The OAuth Authorisation Server wrongly grants tokens to a malicious third party.
An Access Token created for a legitimate Relying Party makes its way into the possession of a malicious entity, enabling it to masquerade as that Relying Party.
The Relying Party receives and trusts a token that was actually created by a malicious entity.

In order to address the first problem, a Relying Party needs to have an existing relationship with the OAuth Identity Provider before it can use OIDC to authenticate its users. This involves the Relying Party being registered with that OAuth Server and having a client ID and a client secret. That identifying information is then included whenever the Relying Party makes a request for tokens, so that the OAuth Server can be sure that the request is legitimate. Of course, this means that that ID and secret need to be stored securely by the Relying Party; for instance, all API calls using them should be made from a server, rather than a user's browser.

The second possibility could come about in a couple of ways; either the Access Token is intercepted in transit, or it's leaked to external parties by the legitimate Relying Party. To avoid the first of these, all the API calls should be made using HTTPS, so that all the traffic is encrypted and can't be inspected by any onlookers. To avoid the second problem, it's important that the Relying Party should always treat Access Tokens securely, and this includes only ever using them in a server and not in the user's browser (just as with the client ID and secret).

The third problem is largely prevented by the fact that ID Tokens are JWTs, which are cryptographically signed by design. If, however, the OAuth Authorisation Server isn't signing its JWTs (which it should!), then this vulnerability can be mitigated by the Relying Party only ever trusting tokens that result from a direct API call. It can also verify the tokens it receives by sending them to the Authorisation Server's /introspect endpoint, which will tell the Relying Party whether or not a given token is valid.

Using OpenID Connect in your web apps

So, what do you need to do in order to enable OpenID Connect authentication for your web apps? Well, you need your web app to be able to make the appropriate API calls to your OAuth Identity Provider of choice (making sure you ask for the right kind of tokens!) and for your app to be able to safely handle the responses it gets. Unfortunately, this can be pretty fraught, because you need to be certain that you're doing everything correctly at every stage of the process, or else you risk falling foul of some of the vulnerabilities we discussed above. However, once it's set up, you get all the benefits of SSO.

Anvil apps come with several ready-made options for using OIDC-based SSO; several OpenID Identity Providers such as Google and Facebook are already configured.

It's nice to know about tokens... but you don't have to.

Enabling SSO for your users is as simple as using Anvil's built-in User Authentication service, and ticking a single checkbox. Easy!

More about Anvil

Try Anvil for free!

Creating a Pure Python Web Dashboard with the QuickBooks API

Eli Holderness — Fri, 25 Mar 2022 15:20:25 +0000

QuickBooks is flexible, powerful accounting software, used by businesses and individuals alike. It also has a robust API - and I'm going to show you how to use it. We're going to build a public dashboard, displaying company revenue by month.

We're building it with Anvil, so we can build and deploy this app entirely in Python (no HTML or JavaScript required!). We're not just driving the QuickBooks API with Python code -- we're building the front-end, including interactive plots in the browser, with Python too!

Here's the finished app:

Open the app here, or read on for a run-down of how it's built!

Here's how we put this dashboard together:

Create our apps and set up credentials
Authenticating and getting API tokens
Calling the QuickBooks API to get our data
Display our data to our users

You can open the app's source code and follow along -- just click below:

Clone the app!

Ready? Let's begin.

1. Setting up our apps and credentials

To get started, we just head to the Anvil App Editor, hit 'New Blank App' and choose 'Material Design'. This gives us a layout where we can start dragging and dropping components, such as a title.

Then, deploying the app is as simple as clicking 'Publish App' in the app's settings, choosing a URL, and hitting 'Apply'.

Now our app is available to anyone on the web - but it's not very exciting yet! To start pulling accounting data from Quickbooks, we'll need to set up an account there too.

We head to the Intuit developer page and sign up there for a developer account. From there, we can create a sandbox company, which is a dummy company that comes pre-filled with some example accounting data.

Once we've got our sandbox company all set up, we go to the Dashboard to create an Intuit app. This is the entity that provides us with authentication details, so that our Anvil app can access our QuickBooks data.

Then, from the Dashboard, we can access the credentials from our Intuit app; we're looking for a Client ID and Client Secret, and they can be found under the 'Keys and OAuth' section.

We store these in our Anvil app as App Secrets, so we can access them later from code.

Now we're armed with the credentials we need, we can get started on the authentication flow for our Anvil app!

2. Getting API tokens

To access the QuickBooks API, our app needs API tokens. We have to get those interactively (via OAuth), but they're then valid for 100 days, so we can store them in our database and use them to fetch data whenever our app is opened. (We're not going to get into a full OAuth tutorial here -- you can see how it works in our app's source code, or read a more detailed guide in the QuickBooks docs).

Once we have tokens, we store them in our app's built-in database, using Anvil's built-in encryption. When the token expires, we'll need to repeat the OAuth login process, but that's pretty quick:

3. Call the QuickBooks API

Now our app has all the credentials it needs to start making calls to the QuickBooks API. We're going to use the ProfitAndLoss endpoint to retrieve data about our total income over the last year, grouped by month.

Here's the server function:

@anvil.server.callable
def get_past_yearly_data():
  # The 'quickbooks_auth' table has one row, with all our tokens in. Fetch it:
  row = app_tables.quickbooks_auth.get()

  # Decrypt the Access Token
  access_token = anvil.secrets.decrypt_with_key('tokens_key', row['encrypted_access_token'])

  # Hit the ProfitAndLoss endpoint
  today = date.today()
  resp = requests.get(
    url=f'https://sandbox-quickbooks.api.intuit.com/v3/company/{row['realm_id']}/reports/ProfitAndLoss',
    headers={
      "accept": "application/json",
      "authorization": f"Bearer {access_token}",
      "content-type": "application/json",
    },
    params={
      'start_date': f'{today.year - 1}-{today.month + 1}-01',
      'end_date': today.isoformat(),
      'summarize_column_by': 'Month',
      'minorversion': "57"
    }
  ).json()

Here we see the code for the API call; we get the access token from our Data Table and decrypt it, then make a GET request using the Python requests package.

The data that comes back is a little gnarly (here's a sample), but it's nothing we can't fix with a few list comprehensions:

  months = [col["ColTitle"] for col in resp["Columns"]["Column"]][1:]
  summary = [col["Summary"] for col in resp["Rows"]["Row"] if col["Summary"]["ColData"][0]["value"] == "Total Income"][0]["ColData"][1:]
  incomes = [item["value"] for item in summary]

  return months, incomes

4. Display our data

Now for the easiest part: Call that get_past_yearly_data() function from the browser, and put the results into a Plotly chart.

We create a Plot component using Anvil's drag-and-drop GUI editor, and call it revenue_plot:

Anvil lets us call server functions directly from browser code. Our server function returns two lists: one with all the months for which we have data, and one with the incomes for those months.

  # This code runs in the browser:
  months, incomes = anvil.server.call('get_past_yearly_data')

Now we can import the Plotly API:

import plotly.graph_objects as go

...and make a bar chart:

self.revenue_plot.data = go.Bar(
    x = months,
    y = incomes,
    marker={'color': '#39bfbc'}
)

self.revenue_plot.layout = go.Layout(
    title=go.layout.Title(text="Last year's monthly income", x=0.5),
    yaxis_title="Income (USD)"
)

And that's it! Here's our finished app:

See the code

If you want to see the source code for this app, you can click here to clone it!

Go even further

Of course, we've only barely scratched the surface of what it's possible to do with the QuickBooks API. Using this framework, you could build a fully featured dashboard and make much more use of Plotly's interactive features. Have a go at extending what we've built today!

Clone the app!

More about Anvil

Try Anvil for free!

A Gentle Introduction to SAML Authentication

Eli Holderness — Thu, 03 Mar 2022 15:29:32 +0000

Implementing SSO with the power of XML

In the previous post in this series we covered Single Sign-On, what it is, and what it means for web services. This time, we’re taking a deep technical dive into one of the two main technologies that are used to implement it: Security Assertion Markup Language, colloquially known as SAML. So what is it, and how can we use it for SSO?

SAML defines an interoperable, standardised protocol for letting a web service (in SAML world, a Service Provider or SP) authenticate a user with an identity provided by an external party (an Identity Provider or IdP). In essence, SSO with SAML allows a Service Provider to delegate its user authentication responsibilities to an Identity Provider. All the communications between the SP and the IdP follow a particular XML format, and SAML protocols can also handle use cases for authorization and identity provider discovery - but, in this blog post, we'll be focusing specifically on the web SSO use case.

SAML from beginning to end

Let's say you're a web developer who wants to be able to use SAML SSO to authenticate your users. What exactly do you have to know?

A typical SAML 2.0 SSO authentication flow goes like this:

The user visits the Service Provider with their browser.
The Service Provider redirects the user to the Identity Provider, along with a SAML Authentication Request.
The user authenticates with the Identity Provider, if they aren't already authenticated.
The Identity Provider returns the user to the Service Provider, along with a SAML Authentication Assertion.
The Service Provider cryptographically verifies that authentication assertion.
The user is now authenticated with the Service Provider.

There are a couple of different ways some of these steps can happen, but the broad strokes stay the same. One interesting property of this flow is that the SP and IdP never actually communicate directly. They redirect the user's browser back and forth along with SAML Requests and Assertions, but they don't actually need to be on the same network, which means you can use an IdP that's on a private corporate network to authenticate with an SP that's on the public internet.

Before the above flow can happen, both the SP and the IdP need to be configured to trust each other, which is done by exchanging some key information between them. In order to get everything set up, you'll need to understand a bit more about what exactly those entities are.

What is a Service Provider?

If you want to use SAML SSO to let users authenticate wtih your web service, you'll have to set that web service up to act as a Service Provider for SAML purposes. Perhaps you're using a development platform with a SAML integration (like Anvil!), in which case this is simple - but, if you're doing it from scratch, here's what you need to know.

The key features of a Service Provider which allow it to interact with an Identity Provider are these:

It has a unique identifier which allows Identity Providers to keep track of it
It owns a signing certificate, which allows Identity Providers to trust the messages it sends
It has a specific HTTP endpoint, which allows Identity Providers to know where to send any replies (including Authentication Assertions)
It needs to be able to send users to an Identity Provider in order to authenticate, and to be able to understand whatever response the Identity Provider returns.

Let's look at the three first bullet points. When configuring a relationship between your SP and IdP, all those pieces of information about your Service Provider need to be given to the Identity Provider you'd like to use. Lots of IdPs provide a neat way to do this: they expect a metadata file produced by your SP, which bundles all this information up. If you're building a Service Provider, you might like to add functionality to construct and download a file in the expected format.

Below is an example Service Provider metadata file (downloaded from an Anvil app), which illustrates how each of those 3 pieces of information fit into it.

<?xml version="1.0"?>
  <md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" validUntil="2020-10-23T14:15:21Z" cacheDuration="PT604800S" entityID="http://anvil.works/apps/_/saml-app/79aac3df685d66" ID="ANVIL_444fcc73">
    <md:SPSSODescriptor AuthnRequestsSigned="true" WantAssertionsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
      <md:KeyDescriptor use="signing">
        <ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
          <ds:X509Data>
            <ds:X509Certificate>
              -----BEGIN CERTIFICATE-----
              MIIE...
            </ds:X509Certificate> 
          </ds:X509Data>
        </ds:KeyInfo>
      </md:KeyDescriptor>
      <md:NameIDFormat>
        urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress
      </md:NameIDFormat>
      <md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="http://anvil.works/apps/_/saml_auth_login" index="1"/>
    </md:SPSSODescriptor>
  </md:EntityDescriptor>

Firstly, within the md:EntityDescriptor tag, there's an attribute called entityID, the value of which is the unique identifier for your Service Provider. Then, further down, we can see a ds:X509Data section, which is where your Service Provider's signing certificate would be placed. (In the above example, the certificate itself is truncated for brevity).

Finally, within the md:AssertionConsumerService tag, we can see a Location attribute, containing a URL. This is the endpoint to which the Service Provider expects any responses to be sent. Within that tag is also a Binding attribute, which tells the Identity Provider that (in this case) the SP expects any responses to that endpoint to be sent via an HTTP-POST request.

Even if an Identity Provider doesn't have an option to upload a metadata file from your Service Provider, it will definitely need to ask you for these three pieces of data in some other way. Without them, it won't know what your service is, why it should trust it, or how to talk to it.

So, once you've defined those three aspects of your Service Provider, you'll need to address the last bullet point on the list above: your SP needs to be able to actually interact with the Identity Provider. That means sending users there when they want to authenticate, and understanding what the Identity Provider tells you about them when they come back. In order to make that happen, we'll need to understand more about what an Identity Provider does.

What is an Identity Provider?

The things that an Identity Provider needs - perhaps unsurprisingly! - mirror those of a Service Provider. Here's what defines an IdP:

It has a unique identifier which allows Service Providers to keep track of it
It owns a signing certificate, which allows Service Providers to trust messages it sends
It has a specific HTTP endpoint, which allows Service Providers to know where to send their Authentication Requests
It needs to be able to receive Authentication Requests from Service Providers, handle those requests (including authenticating users), and send a reply that the Service Provider can understand.

Just as with Service Providers, it's typical for the three pieces of information above (identity, certificate and endpoint) to be bundled into a metadata file. The main difference between the two types of metadata file is that rather than an AssertionConsumerService URL (which is where a Service Provider consumes Authentication Assertions), an Identity Provider will have an SingleSignOnService URL (where it consumes Authentication Requests for the purposes of SSO).

As mentioned above, there are a couple ways configurations can differ - for example, the IdP can expect the user to be sent over with either a HTTP-GET or HTTP-POST request, or sometimes either - and all this information would also be expressed in the SP's metadata document.

Authentication Requests and Assertions

With the above information successfully exchanged, our Service Provider and our Identity Provider know who each other are, how to trust each other's messages, and where to send any messages of their own. Great! So what's actually in those messages?

To kick off the process of authenticating a user with SAML, a user visits the Service Provider's website or app (step 1 in the flow above). The Service Provider will then send that user to the Identity Provider along with an Authentication Request (step 2). This Request is contained within an XML document, and that document is then sent as a query parameter on an HTTP request (made from the user's browser) to the Identity Provider's SingleSignOnService URL. This Authentication Request can contain information such as:

The identity of the Service Provider which is making the request
What kind of authentication the SP wants the IdP to perform (for example, whether it should require a password from the user who's trying to authenticate)
Whether the user should be allowed to create a new account when they arrive at the IdP (thus allowing a new user to authenticate with the Service Provider)

The Request is contained within a larger XML document which should also include a Signature section; that sections then contains a signature over the Authentication Request, generated using the Service Provider's signing certificate. This signature allows the Identity Provider to verify that the request was sent by the Service Provider and that it hasn't been tampered with in transit. All this would be pretty standard public-key cryptography procedure if the payload were a byte-string rather than XML, but - as we'll discuss later - multiple byte-strings can represent the same XML data, and this can introduce complications down the line.

The Authentication Assertion that the Identity Provider sends once the user has authenticated is very similar; it's sent (also via HTTP) to the Service Provider's AssertionConsumerService URL, and it can include the following:

The identity of the IdP
The identity of the Service Provider for whom the Assertion is intended
The identity of the user who has authenticated, and optionally some attributes that the IdP has stored about them (for example, name or email address)
How that user authenticated (for example, as above, using a password)
The conditions under which the Assertion should be considered valid (for example, only within a certain time window)

Just like the Request made by the Service Provider, the Assertion is contained within a larger XML document, which also contains a signature over that Assertion.

When the Service Provider receives that Authentication Assertion and verifies its signature, it can be confident that the user has successfully completed the required login flow, and safely let them access login-restricted resources. You're done!

SAML and security

SAML's security posture is as follows: when the SP and IdP are being configured to trust each other, part of the data exchanged between them is access to each other's X509Certificate - a certificate used for public-key cryptography. Typically, whenever a SAML message (in XML format) is then sent between the two parties during an authentication flow, that XML will be signed by the sending party, using their private key. The receiving party will then be able to use the X509Certificate that they have been given in order to verify that that SAML message is definitely from the sending party and that the data hasn't been modified during transit.

Over the years, there have been a lot of vulnerabilities found in SAML systems, of various kinds. Most of them stem from the fact that the entire framework is based on XML, which is optimised for flexibility rather than a single robust path.

Let's take a look at one particular way that that flexibility can introduce vulnerabilities.

Signatures and Canonical forms

As mentioned above, SAML uses what sounds like fairly standard public-key cryptography - but there's a complicating factor: signing XML data is hard.

The fundamental problem with signing XML is that two different XML documents could represent the same information. For example, look at these two tags:

<saml:Issuer>http://sp.example.com/demo1/metadata.php</saml:Issuer>

  <  saml:Issuer  ><http://sp.example.com/demo1/metadata.php</  saml:Issuer  >

These mean the same thing, but if you consider them as two series of bytes, they're different and would therefore result in different signatures. During the parsing and handling of XML data that needs to happen during a SAML authentication flow, it's entirely plausible that two different legal representations of the same data might emerge. There needs to be a way of distinguishing when two XML documents are 'really' the same; that's what canonicalisation does for us.

However, this is both complicated and difficult. In fact, the SAML specifications lay out three different algorithms for it! Again, this flexibility increases the potential for vulnerability.

Flying under the radar with XML comments

A few years ago a new vulnerability was also discovered, which allowed an attacker to masquerade as a fully authenticated user. There's a great write-up of it here, but the long and short of it is as follows:

Not every XML parsing library handles things such as comments consistently, and some canonicalisation algorithms ignore comments. This allows for a malicious user to use comments to alter various aspects of the XML request that gets sent to the Identity Provider. In particular, they could affect the way that the Identity Provider parses the identity of the user who is trying to authenticate.

But wait! It gets worse: the cryptographic signature is generated over the canonicalised version of the document. Because, in this case, that canonicalisation is ignoring comments, this means that even the "tamper-proof" cryptography won't pick up on the fact that the assertion the attacker is presenting is not what the IdP signed. Thus, an attacker can present an apparently valid signature over an assertion for an arbitrary user, and authenticate as them.

These days, many SAML implementations have addressed this vulnerability by switching to XML parsing libraries that handle comments in a safe way, and introducing checks against this specific kind of attack.

In general, SAML is widely used enough that it's had quite a few eyeballs on it, and - as is often the case with SSO technologies! - if it breaks, you at least won't be the only one in hot water. However, if you're implementing your own SAML system, there are plenty of ways you can leave yourself open to attack.

What are the drawbacks?

These security concerns are worth taking seriously; if you're building logic for a Service Provider, you'll need to use a SAML library that addresses them (and keep your eyes peeled for any new vulnerabilities that are discovered). If you're writing a SAML or XML-parsing library, then you definitely need to make sure you understand and guard against the kind of attacks described above!

Another obvious drawback is the amount of overhead involved in using SAML for SSO. In theory, it can be as simple as downloading two metadata files and uploading them in the right places, but not all SPs and IdPs let you do this quite so easily. If that's the case, you'll have to understand SAML pretty well in order to get things off the ground. This is one of the reasons for the development of OpenID Connect, the other major technology used to implement SSO, which we'll be covering in the next blog post in this series.

Of course, none of these drawbacks will affect you if you want to enable SAML SSO for your Anvil apps; we've got out-of-the-box, up-to-date SAML integration. It's a single click to add to your app, and setting up a SAML relationship with your Identity Provider of choice is as straightforward as can be. Check out our documentation to learn more.

More about Anvil

Try Anvil for free!

An SSO primer: what is it, and how does it work?

Eli Holderness — Tue, 01 Mar 2022 17:50:08 +0000

Too many logins spoil the online experience

Imagine you run a tech company. Your employees have got a bunch of different services that they want to use - a ticketing system, an internal wiki, an online code store, an HR system - the list goes on. These are all genuinely different services, perhaps owned or managed by different entities, so you can't just lump them all into one big app. That's a lot of accounts per person - and you have to configure them for each new person you hire, every time someones changes their name, or whenever someone leaves the company. If you miss a single account at any of these stages it could be a serious problem. Plus, it's a pain for your employees to have to sign into all of these places every time they clock in for the day. Surely there's got to be a better way!

What if they only had one account, which let them log in to any of these services? Then, each admin action - a name change, a password reset, setting up or closing out accounts - would only need to happen once, in one place. Suddenly, you have a unique and consistent identity for each employee, and a single source of truth for all their details. And, as a bonus, your employees don't have to remember 6 different passwords and jump through a set of login screens every morning.

That's the idea behind SSO (which, appropriately, stands for Single Sign-On).

How does it work?

SSO depends on the following principle: there is some central entity that manages your identity, and is able to verify that you are who you say you are. Then, other services delegate their authentication to that central identity. The authentication flow goes as follows:

You visit your web service of choice.
That web service sends you to your SSO identity provider's website, where you log in (if you're not already logged in).
The identity provider sends you back to the web service, along with some token to prove your identity.
The web service derives your identity from that token (either cryptographically, or just by asking the identity provider).
The web service then trusts that you have rightfully authenticated as that identity, because they trust the identity provider.

In order to set this up, a relationship between that web service and the identity provider needs to be established before the web service will trust the identity provider to authenticate users, and so that the identity provider knows what's going on when it gets an authentication request from the web service.

Once you've set up a relationship between your web service and a central identity provider, any user that has an identity managed by that provider can use it to log in to your web service. If they're already authenticated that identity provider, your web service will be able to authenticate them without any active input from the user. If the user isn't already authenticated the identity provider, they'll be prompted to do so, and this authentication is the aforementioned Single Sign-On.

By this mechanism, one identity managed by one service can be used subsequently to authenticate the user with multiple services - without needing those services to store or verify user credentials directly. Convenient!

What are the benefits?

Obviously, it massively reduces user friction. Constantly having to log in to services is a pain, and using a central provider to manage identities means you don't end up with a proliferation of passwords to forget.

On the business and administration side, it means a lot fewer password reset requests for the IT department to handle, and when somebody needs to change some aspect of their identity - for example, a name change - it only needs to happen in one place. When a new user joins the organisation, or an existing one leaves, there's only one account to worry about, which reduces administrative overhead and the opportunity to leave stale accounts hanging around as a security risk.

When it comes to security, you might think that a single set of credentials is far riskier than multiple - after all, they only need to get stolen once, right? However, when done right, SSO can actually reduce risk. By removing the need for multiple sets of credentials and abstracting the process away from the user, it decreases the viability of several common attacks; for example, imagine one of these web services experiences a security breach. You now don't have to worry about someone stealing your users' passwords and attempting to use them elsewhere, because that web service never saw or needed your users' passwords in the first place.

SSO also makes two-factor authentication much more attractive as a prospect, since you only have to do it in one place to see the security benefits across all your services. Moreover, if you delegate your identity management to something like Google, you can rely on their security engineering budget rather than yours. This means getting access to security features like 2FA (with minimal setup), and risk-based authentication - which smaller service providers will be hard-pressed to set up.

SSO isn't a technology, it's a goal

This is all a bit abstract, and that's because SSO, as a standalone concept, is just that; a concept. In my next couple of blog posts, we'll dive into the two most popular technologies used to implement it - SAML and OIDC - and take a closer look at how they work.

Anvil apps come with several ready-made options for SSO - you can configure a SAML-based relationship with any Identity Provider with no fuss, and several OpenID Identity Providers such as Google, Facebook and Microsoft are already configured.

More about Anvil

Try Anvil for free!

DEV Community: Eli Holderness

Using Coiled with Anvil

What is Coiled?

What is Anvil?

How can we use the two together?

Configuring your Anvil app's server environment

Using Background Tasks

Building a UI

Wrangling some data

Plotting our data

Generating PDF output

Wrapping up

Going even further

More about Anvil

Using custom Python packages with Anvil

Step 1: Create your Anvil app

Step 2: Install packages into your app's server environment

Step 3: Write a function to generate word clouds

Getting a random Wikipedia page

Step 4: Build your app's UI

Step 5: Writing client Python code

Clone the App

New to Anvil?

Build a Morse Code Generator with the Pico W

Flash Morse Code Messages on the Pico W - with an interactive, secure, authenticated web app!

Getting started

Showing Morse code with your Pico's LED

Translating text into Morse code

Building a user interface for your app

Putting it all together

Adding user authentication

Adding the Users Service

More about Anvil

Pointers? In My Python? It's More Likely Than You Think - Part 3: Object Lifetimes and Garbage Collection

How CPython can tell when you're done with an object, and what happens next

The __del__ magic method

Reference counting in CPython

Collecting garbage in CPython

Detecting cyclic isolates

Finalisers behaving badly

So what have we learned?

See this article as a talk

More about Anvil

Get Started with the Pico W

Build your first internet-connected device

Setting up your Pico W

Building your Anvil app

Creating an Anvil app

Connecting your Pico W to your web app

Adding your WiFi network details into boot.py

Adding your Uplink key into main.py

Understanding the main.py file

Updating your app to call your Pico W

Publishing your app on the web

Go even further

More about Anvil

Pointers? In My Python? It's More Likely Than You Think - Part 2: Equality

Object IDs, and why they matter

is uses id(x)

== uses __eq__

More about Anvil

Building A Python Code Completer at PyCon US 2022

More about Anvil

Pointers? In My Python? It's More Likely Than You Think

A tour of Python's memory magic

What pointers are, and where you'll find them

Pointer aliasing

Immutable Objects (or: Tuples Behaving Badly)

The += operator

More about Anvil

OAuth and OIDC: What You Need To Know

OIDC under the microscope

What is OAuth?

... so what does it do?

How does it work?

Why do we need something else for authentication?

OAuth isn't unusable for authentication - but we can do better

What is OpenID Connect?

OIDC and security

Using OpenID Connect in your web apps

The `del` magic method

Adding your WiFi network details into `boot.py`

Adding your Uplink key into `main.py`

Understanding the `main.py` file

`is` uses `id(x)`

`==` uses `eq`

The `+=` operator