DEV Community: Tom Larkworthy

A Custom Domain in 4 lines of code

Tom Larkworthy — Thu, 17 Mar 2022 18:22:20 +0000

When clients ask me to add a vanity domain, I usually use their Cloud provider, which is ok but tends to be a little fiddly and not particularly cheap.

For a private project, I wanted a custom domain ASAP, for free. I think I found my simplest setup using Netlify rewrite rules. The beauty of Netlify is that it can sync with a git repository and so the only native tool you need is git.

So all I needed to do was commit the following netlify.toml to a Github and then deploy my custom domain from Github.

[[redirects]]
  status = 200
  from = "*"
  to = "https://webcode.run/observablehq.com/@tomlarkworthy/tarot-webserver:splat"

Setting up Netlify to handle a custom domain is one of the easiest processes, even if you register the domain elsewhere.

This 1 hour of work the first time you do it and less than 15 minutes of work going forward. Custom domains with SSL could not be easier.

Cover Photo by mk. s on Unsplash

WEBCode.run Private Endpoints on Observablehq.com Released!

Tom Larkworthy — Tue, 18 Jan 2022 15:27:20 +0000

WEBCode.run Private Endpoints Released

WEBCode.run is growing! We have added our first paid feature which will help support it long term. WEBCode is bootstrapped so this is a big milestone!

For those that do not know, WEBCode.run allows you to run serverless compute workloads from within Observable notebooks. Until now, notebooks containing WEBCode.run handlers had to be public so that WEBCode's compute layer could read the code. Not anymore!

We have added a new feature for Observable Team accounts that allows public access to serverless endpoints while keeping the source code private. We feel this feature best suits commercial teams, and thus, is a perfect fit for a paid tier. Find out more on the webcode.run website or contact sales@webcode.run.

For those that want to use WEBCode to build transparent open source services, they can continue do so at no charge.

As part of the work in providing a paid tier, we have updated the website significantly. You can also email me at tom@webcode.run!

Coming soon

WEBCode.run enables running HTTP services with excellent performance and observability from within a notebook. In the coming months we will demonstrate why this is an important primitive. WEBCode.run is a connection technology for something larger, it enables completely self documenting projects to be bundled into a single encapsulated artifact. In the coming months the focus will be on delivering preconfigured artifacts you can one click fork to self-host and customize.

In our vision of the future, source level customization and self-hosting at the service level is a single click away. This is only possible when the backend and frontend + all the extras (documentation, monitoring) are a single forkable artifact, WEBCode.run with Observablehq.com are able to deliver this.

Support us on Product Hunt

We are launching the paid tier and webcode.run on Product Hunt. Help us get to #1 with some love here

Follow the newsletter

I am going to start posting webcode.run updates to a newsletter, so this is another option you can stay up to date

100 Beautiful and Informative Notebooks of 2021

Tom Larkworthy — Fri, 31 Dec 2021 10:33:08 +0000

It's the end of 2021, and I would like to reminisce about the amazing work that was produced on Observable over the last year. I have taken all the notebooks that were trending from the Top Trending Notebook Twitter Bot and picked 100 good ones.

The broad categories apparent in 2021 were "dataviz, art, mathematics, maps, apps, development, design, tutorials and tools". Every notebook listed here is worth a read, but there is rather a lot so it might take you a few sessions!

OK, enough chat... here are my favorite notebooks of 2021, sorted into topics...

Dataviz

Observable is a digital community for data practitioners. In 2021 an enormous quantity of informative dataviz notebooks were produced, but here is some of the very best. Special shout out to @karimdouieb who has been on fire using 3D graphics to visualize large datasets with great effect in works like All the passes.

The Sun’s analemma by @fil

If you were to look at the sky every of the year at the exact same hour, where would the Sun be? The diagram below shows its position for each hour of the day, limited by the horizon. Each twisted loop (called an analemma), is in fact made of dots, one for each of the year. The analemma shows by how much solar time is fast or slow compared to clock time, at different periods of the year. Click on the map to set the observer’s location (defaults to Greenwich).

Equal-Area Radial Matrix of LGBT Rights by @kerryrodden

In Why use a radial data visualization?, I wrote about some of the pros and cons of choosing a radial layout for time series, categorical, or hierarchical data. I left out matrix/heatmap data, though—so for Pride month, here’s a partial re-creation of a radial matrix visualization I love, The Guardian’s Gay rights in the US, state by state (whose scope covers both sexual orientation and gender identity, despite the title).

All the passes ⚽️ by @karimdouieb

A visualisation of 882,536 passes from 890 matches played in various major leagues/cups such as the Champion League 1999 FA Women’s Super League 2018 FIFA World Cup 2018, La Liga 2004 – 2020 NWSL 2018 Premier League 2003 – 2004 Women’s World Cup 2019 Data provided by StatsBomb Original inspiration: Alexander Varlamov's blog post Data Dependencies

State of Dataviz by @observablehq

2021 Observations: Data through the lens of the Observable Community One of Observable’s motivating principles is to listen to and learn from its passionate and fast-growing community. Hundreds of our community members generously participated in two research surveys about data analysis and visualization over the past year. Here’s what we learned. Data is exploding, for everyone With the explosive growth in online technologies, services and information, data is overwhelming for all of us.

Women in space 👩🏼‍🚀 by @ralucanicola

Data I parsed the data from the Wikipedia site List of female spacefarers. I manually enhanced the comments section with information from individual Wikipedia profiles of the astronauts and information from the NASA site. Sky image by Max McKinnon on Unsplash.

Day 9: Smoke Basin by @visnup

These caves seem to be lava tubes. Parts are even still volcanically active; small hydrothermal vents release smoke into the caves that slowly settles like rain. – Advent of Code 2021, Day 9 Part 2 Find the three largest basins. Starting from a low point (or really, any point in a basin), breadth-first search out from it visiting all its neighbors and keep going, stopping at any 9s. Use a FIFO queue(https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics) and when it’s exhausted we’re done.

EU Fishing Flow by @benjamesdavis

Data Helper Functions Scales Dependancies

What story do two decades of weather data tell you? by @analyzer2004

What story do two decades of daily weather data tell you? You are about to find out 😊. As usual, I picked these five major cities across 1000 miles of the US West Coast. The top part of the graph contains two stacked column charts. The left one represents the temperature distribution of the current month, and the right one shows weather condition distribution.

Visualizing Seasonal Daylight by @dbridges

Everyone who lives away from the equator knows how the days lengthen and shorten throughout the year. We hear about the long dark polar winters, and the endlessly looping sun of the polar summer. Many people might be able to toss out the answer to these natural phenomena, "because the Earth is tilted," but in my experience far fewer intuitively grasp the geometries involved.

How much warmer? (BBC) by @fil

A recreation, for educational purposes, of this beautiful map by the BBC Visual and Data Journalism team (31 July 2019). You can learn more on the original project in this very detailed and complete write-up by the BBC team. Made with D3 (the original visualization uses THREE.js). See also Matt’s version, which uses a phyllotaxis distribution instead of the original’s rectangular grid.

Votes for Women: Suffragette Collage Maker by @oliviafvane

Submission for Observable's Women's History Month DataViz Contest. Thank you to Jacob Rus for his help optimising the code. Political movements generate their own “stuff”: the posters from the the 2017 Women's March (and the Pussyhat), the Brexit tea towel, the gilets of the Gilets Jaunes. The same is true when women were fighting for the vote in the 20th Century.

2021 Observable Community Recognition by @observablehq

Observable is built as a space where those who work with data can come together to explore, analyze, discover and communicate with each other. Working with data and making sense of it is challenging, but with the contributions of you, the community, it is easier than ever to work together to make sense of the world with data. Our community inspires data collaborators around the world to collaborate, share, learn, and teach. Every day our community invents new and creative ways to explore and visualize data.

The Most Frequently Used Emoji of 2021 by @jenniferdaniel

92% of the world’s online population use emoji — but which emoji are we using? Well, it appears that reports of Tears of Joy’s death are greatly exaggerated 😂. According to data collected by the Unicode Consortium, the not-for-profit organization responsible for digitizing the world’s languages, Tears of Joy accounts for over 5% of all emoji use (the only other character that comes close is ❤️ and there is a steeeeeep cliff after that). The top ten emoji used worldwide are 😂 ❤️ 🤣 👍 😭 🙏 😘 🥰 😍 😊.

Hacking Power BI Charts with Plot by @analyzer2004

Are you a fan of both PowerBI and Observable Plot? Here are some ideas for recreating PowerBI charts with the 🙏almighty Plot🙏!! Waterfall Chart Ribbon Chart It's a variant of Bump Chart which reflects both value and the changes in rank over time. Tornado Chart LineDot Chart Waffle Chart Funnel Chart 🌐ericlo.

Lucy mission animation by @theeconomist

This notebook contains the code for generating the animated diagram of NASA’s Lucy probe, which will visit many of the “Trojan asteroids”, a pair of clusters of asteroids at the Lagrange points of Jupiter’s orbit. You can read more about Lucy’s mission and see the complete animation in our article. The visualisation above shows one frame of the animation. Labels were added to the final video in AfterEffects, so this diagram has none.

The story of Observable on Twitter by @floatingpurr

** Main info** 📝 forked integration notebook: Google Sheets Starter ℹ️ dataset: the Observable twitter activity up to (and updated to) 🗓 (source: Twitter API) 📗 Data are hosted in a Google Sheet and published as a csv file Once Upon a Time... ✨✨✨ 💫 Welcome to the data story of Observable on Twitter. Every story has a beginning... Do you remember this? Back in 2017, that was the first "official" Observable tweet, and it has been liked times 💙 (as of 2021-07-26).

Digital and Generative Art

The Observable platform is great for iterating on visuals quickly. In 2021 there were many digital artists producing novel and interesting works, some of my top picks are shown below. Makio135's Creative Coding tutorial notebook on producing generative art on Observable remains a must-read for aspiring digital artists.

Flow fields by @esperanc

An attempt to reproduce some of the techniques described in Tyler Hobbs' Flow Fields post. A simple field A more interesting field Curves through the field Position sampling Perlin noise fields Colored Perlin fields The idea here is to set the color of each flow curve to the color picked from a "seed" image.

Simplex Stacks by @enjalot

You may have heard of Simplex Noise, you can find many amazing visuals using it on Observable. But have you ever seen a stacked area chart of Simplex noise? This notebook does just that, using Plot to visualize the noise in a number of familiar and unfamiliar ways. Try changing the z\ parameter to see the noise in action: The data samples the simplex function primarily along the x axis. Each series in the stack is sampled at a single y value.

Procedural Face Exploration by @stringertheory

This is a copy, with only very slight modifications, of the excellent Procedural Face Generator written by Amit Patel. I wanted to play around with some of the facial features so I ported it into Observable, but you should check out the original (especially the cool sliders that help understand how changing the parameters will change the face 😍)! It's interesting to explore how the face color, eye shape, and the addition of eyelashes change the way the face looks.

Pseudo-Blue Noise by @jobleonard

Blue noise is a noise color[1] that is particularly well-suited for image dithering purposes (see also [2][3]). Unfortunately it's also computationally intensive to generate. For example, Bart Wronski mentions that generating a "64×64 wrapping blue noise-like sequence" took "a couple hours on an old MacBook." This was in 2016, but unless dramatic progress has been made in the algorithms to generate blue noise this will not be significantly faster today.

D3 Force Along a Path by @mootari

Implements a custom force for d3-force which moves nodes along an SVG path. "Track strength" vs "Max step size" At each simulation step ("tick") a node is moved towards its target position on the path. The further a node is from its target, the more distance it will cover during a tick.

The "Track strength" multiplier (strength\ option) gets applied to this distance. A value of 0.5\ means that the node will cover half the remaining distance towards its target on each tick.

Generative Painting by @karimdouieb

This is a recreation of a generative water colour like painting technique described beautifully by Tyler Hobbs during his "How to Hack a Painting" talk in the Strange Loop Conference '17. Also checkout his essay on the same subject on his blog.

Building a Parametric Seashell by @bronna

The clam and the conch seem to be drastically different creatures. One spends its life burrowing in the sand, staying in one spot to glean nutrients from the water. The other constantly vaults itself around the seafloor to find its next meal. One has a gently curving, sand-colored shell, whereas the other has a shell spiking and spiraling and splaying out in the colors of the sunset. These two species are opposites in many ways, yet the same equation underlies both of their shells.

SVG flowers by @daformat

I stumbled on this tweet by @wks_jp and since spring is coming, I was inspired to create the same kind of floral patterns using svg stroke-dasharray\. There are two patterns that we randomly draw: Variable density dotted path: a stroke-dasharray\ is used with a 0 length for the stroke so it becomes rounded to evenly spaced circles when stroke-linecap\ which is set to round\. Additionally, using this pattern, the resulting flower may or may not have a solid center drawn.

Processing https://10print.org/ (p5.js) by @hellonearthis

Get started with creative coding. Learn more in the tutorial.

What's That Noise? by @ambassadors

A closer look at various types of 2D noise and how to massage their output. Move your cursor over a noise field (or click/tap) to update its line. (Note: currently not optimized for narrow screens. If you're on mobile, please switch your phone to landscape.

Guided Tour of an Infinite Sphere Grid by @mast4461

An Exploration of Parameter Space Best viewed in fullscreen landscape mode, if your GPU can handle all the ray marching. Shortcuts: I've split the fragment shader into small chunks so that I can move the canvas cell next to the chunk I'm working on, to see both the canvas and the code. This would otherwise be difficult when working on GLSL code in the middle of a large cell.

Playing with Hilbert by @zekenie

Space filling curves are fractals that weave through every point in space. They're useful when you are building a system that needs to

<twilight-zone-voice>\
jump dimensions
</twilight-zone-voice>\. Let's say you have a piece of string that represents 1-dimentional space. You could mark that string with a sharpie and represent a point.

Raytracing in 40 minutes, the Observable port by @migueldeicaza

This is an ObservableHQ port of the "Raytracing in 40 minutes" from Peter Shirley, one cute feature is that when you update any of the values of this port, it automatically updates all the renderings as you change the functions and variables in this file. Peter announced that he would be doing a short lecture explaining this on his twitter feed. 3D Vectors and Colors in this notebook are represented as arrays of three values, there is no special data type for it.

Mathematics

There is something about the reactive interactivity of Observable that makes it useful for intuition building. There is an avid group of mathmaticians on Observable producing engaging mathmatical teaching materials. Prof. McClure
in particular has an extensive set of notebooks to support his classes.

Hilbert Curve Plot by @enjalot

A simple representation of the Hilbert curve, calculated with d3-hilbert and rendered with Plot. Stacking Plots When you stack several orders of Hilbert curves on top of each other the pattern between orders can be a bit clearer. It helped me to notice the multiples of 4, and how each level "up" creates 4 corners of a box surrounding a point at the level below.

Aliasing by @cscheid

This tweet shows the phenomenon of aliasing through a very effective animation. The notebook port here provides a little bit of interactivity so you can explore what happens when a signal is not sampled often enough in a reconstruction: A well-sampled signal must be sampled at least twice as often as the highest frequency in the original signal. It's not obvious yet always possible to reconstruct a signal exactly if it's been sampled at a sufficiently-high rate.

Wrap Your Eye Around Rubik's Cube 👁 by @mast4461

Controls: A technique called ray marching is used to render the cube. Usually in ray marching, rays project out from a single point. Here, we don't just project rays in different directions, but we let them start at different points! The graphic below illustrates how the ray origins and ray directions change with the wrap factor. Scroll it to wrap! Implementation Notes to self: Here's a nice example of how to color objects in ray marching: https://www.shadertoy.

Density contours with linear binning by @d3

new in d3-contour@3 and d3@7! This brownian motion system with particles is visualized with the contours from d3.contourDensity. The first chart uses the traditional nearest neighbor method; the second chart shows the improved method—linear binning—adopted by d3-contour@3. Density contours are calculated in three stages: first, each of the data points registers its weight onto a grid. The total weights on that grid are then blurred, and contours are computed.

Dimensionality Reduction drawings by @saehrimnir

UMAP and t-SNE are iterative dimensionality reduction methods. The picture gets drawn by drawing the voronoi cells of the intermediate results above each other with some opacity.

Hexagon grid rounding by @jrus

If we have points in a hexagon grid represented using “axial coordinates” – and coordinates aligned with two grid axes oriented 1/6 turn (60°) apart – we can round to the nearest hexagon with the pseudocode below: def axial_round(x, y):
xgrid = round(x); ygrid = round(y)
x -= xgrid; y -= ygrid # remainder
if abs(x) >= abs(y):
return xgrid + round(x + 0.5*y), ygrid
else:
return xgrid, ygrid + round(y + 0.

Maps

Maps are a specialised subset of dataviz dealing with realworld spatial data. Maps are a loved topic on Observable. In 2021 @floledermann produced the projection notebook to end all projection notebooks.

Local Spatial Autocorrelation Interactive Tutorial by @michelleeesi

Text in this notebook is adapted from Luc Anselin's GeoDa workbook, available here. GeoDa is a free and open-source program for exploratory spatial data analysis and basic spatial regression modeling that you can download here. When we look at maps, we spot concentrations and groupings of high and low values, start to recognize patterns, and construct stories about what might be going on.

Map Projection Playground by @floledermann

Adapted for my course "Cartographic and Geodetic Foundations for Planners". You can find even more static examples map projections in this overview. Map Style: ⚠️ This causes rendering errors for some projections: For some interrupted projections, you can set the number of lobes: 1 The circles for visualizing distortions are taken from here.

Aires d’accueil des « Gens du voyage » by @visionscarto

Export officiel, par @visionscarto, des données collectées par William Acker dans le cadre de l’inventaire critique publié sous le titre « Où sont les “gens du voyage” ? ». URL de référence : https://visionscarto.net/aires-d-accueil-les-donnees (Date de mise à jour : ) \* Les aires d’accueil sont classées par grandes caractéristiques : aires d’habitat, de petit passage, de grand passage (GP).

Five Oceans by @visionscarto

30DayMapChallenge — Day 18. Water. Attached to this notebook is a topojson file that I crafted more or less by hand. As I was trying to polish my “Born in Antarctica” notebook, I wasted quite a bit of time in an elusive search for a simple shapefile of the Antarctic Convergence. Unable to find a base map of the Oceans, I decided to create this one and add it to my toolbox.

The Earth-Nose-Direction game by @ambassadors

How many friends can you find? SHARE! https://bit.ly/earth-nose-game code starts below Acknowledgments Notebook developed and edited by: @fil @mootari @radames Thanks for the fantastic feedback/ideas from all our ambassadors friends who almost broke their noses/necks testing the notebook.

Getting routes with OSRM by @neocartocnrs

The distance between and is km. It takes hours to cover this distance by car. OSRM (Open Source Routing Machine) is a C++ implementation of a high performance route search engine to obtain the shortest paths in a road network. Available under a simplified BSD license, OSRM is an open source service. See http://project-osrm.org/docs/v5.24.0/api/ Preliminary remarks: To query the OSRM API, you need to have CORS enabled. For that, you can use CORS Anywhere Observable.

Latitude and longitude precision by @mourner

How many decimal digits do you need for longitude and latitude? Following the famous XKCD comic on coordinate precision, let's see how this looks on a map. There's a location defined at the bottom of this page, but we've rounded the coordinates to integers, losing all precision. Now it looks like ** , **. Where could it originally point to? The map below shows the exact area of all locations that round to these numbers. So, with zero decimal digits, we can place the location somewhere around London.

Deriving Pantelleria by @visionscarto

30DayMapChallenge — Day 21. Elevation. DEM This notebook starts with a Digital elevation model (DEM) of the Italian island of Pantelleria, halfway between Sicily and the coast of Tunisia in the Mediterranean Sea. I’ve never been there, but it looks like the perfect shape of an island… anyway… let’s begin. I captured this DEM on nextzen’s terrarium. The DEM is pixels wide by pixels high. For reference, it corresponds to the following projection (we won’t use it in the notebook).

30 days and as many maps by @visionscarto

A look back on the #30DayMapChallenge, 2021. Embarking on this challenge at the last minute was a bit stupid, but it felt like an opportunity to finish a few open projects, to try a few ideas, and to showcase a part of my usual toolbox. Day 1. Points. “Dotted Oceans” This globe uses: a backface projection that allows to “see through” the Earth, and a nice-looking random point distribution. Although it’s a very simple notebook, it was quite successful. Day 2. Lines.

Download the shape of a country by @severo

Select a country (or Antartica) #### Settings You think France is weird? Maybe look at Map Template France Lambert-93 or d3-composite-projections instead. Also, the precision is very bad for small countries. Credits Developed by Sylvain Lesage Data from Natural Earth. Code from Countries by area by Mike Bostock.

PHYLLO FORK 🌻🍴 How much warmer? (BBC) by @mattdzugan

This is a fork of Fil's notebook. I simply replaced the regular lat/lon grid with a spherical Phyllotaxis sampling. How much warmer? (BBC) A recreation, for educational purposes, of this beautiful map by the BBC Visual and Data Journalism team (31 July 2019). You can learn more on the original project in this very detailed and complete write-up by the BBC team. Made with D3 (the original visualization uses THREE.js).

Hello, spatialmerge by @chrispahm

spatialmerge is a (relatively) fast library for spatial joining and merging data in JavaScript 🚀 This notebook will walk you through different ways to combine spatial datasets. It is a (blatant) copy of the "merging data" tutorial from geopandas, and uses the same underlying datasets. Merging data There are two ways to combine datasets in spatialmerge – attribute joins and spatial joins. In an attribute join, a GeoJSON feature collection is combined with an array of objects (e.g.

World Grids by @neocartocnrs

Data Import 2. Map Layout ** 3. ### World Elevation Line Map by @neocartocnrs

Data import. We do... ** 2. Data Handling & stuffs for cartography*. Here, we... 3. Lines * 3. ### Macromap by @oliviafvane

An interactive ‘small multiples’ visualisation for historical map collections. How can we get a handle on very large, partially-digitised collections of maps? We need a map of the maps. Macromap is designed to help researchers understand what map sheets the British Ordnance Survey (OS) made, when and where. In addition, it illuminates what’s been digitised: a lot, but not everything.

30 Day Map Challenge: Day 8 - Blue by @ambassadors

The Ocean blue This literally took about 15 minutes to build, in spite of the fact that I never used the drop shadow technique before.

Let's make an extruded map by @neocartocnrs

Data Import & handling **2. ### Day 11: 3D, 30 Day Map Challenge 2021 by @ambassadors

https://data.linz.govt.nz/layer/50768-nz-contours-topo-150k/ This is a little slow load as it needs to extract the data files from a zip file. The contours Shapefile contains two types of geometry types, LineString and MultiLineString The Mouse can interact with the maps orientation. Add colours based on elevation, there are zones over a range of to ✅ The objective here is to convert the contours of Taranaki into a three.js objects. ✅ Fix map orientation.

Apps

At its core, Observable is a front end development platform, and a few of the community have built fully fledged web applications.

Sign a PDF and Adobe: Go F* Yourself by @tomlarkworthy

Allows super-imposing an image of your signature on a PDF. ** Note: you can also do this using Mac Preview, see Tools > Annotate > Signature (docs)** Why not use Adobe? I once had a Adobe subscription exactly for this purpose but I was unable to cancel it. So it cost me $200 to sign a document. They are predatory, and its not just me. Why not use the free tools on the internet? I want to sign financial documents and privacy is critical. Unfortunately, I cannot audit most free software on the internet.

Morse code trainer by @visnup

Use the button below to practice your Morse code. Sound on 🔊. Once you’ve focused the button by pressing it, you can also use the spacebar. If you’re finding it difficult to encode a dah (a.k.a. dash −), try adjusting the dit (a.k.a. dot ·) duration. Built with @tophtucker while streaming.

Sequencer by @tomlarkworthy

The sequence connects to the drum pads which connect to the samples.

Development

Building complex applications on Observable requires all kinds of supporting libraries. A number of development notebooks stood out in 2021. Databases were a strong theme over the year, due to additional SQL support provided by the platform.

Interactive exploration with DuckDB: 5.2 million rows of college degrees in the browser by @bmschmidt

I've looked in the past at in-browser analytics of a granular database of college degrees using basic Apache Arrow functions and arquero. This notebook does the same thing using the new duckdb wasm module. At first pass it's a cut above in speed and flexibility, because: It can load from compressed parquet files, which allow it to work with larger datasets, faster; It can often be about 10x faster than equivalent in-browser solutions if the data is more than several thousand rows.

Introducing SQL with DuckDB in your browser by @cmudig

SQL is a declarative programming language transforming and querying data. It is declarative, meaning that you specify the data you want and not how the query should be computed. SQL is a great fit for a range of data transformation tasks including filter, sample, aggregation, window, and, join. SQL is the most popular query language for database management systems and essential to many business and data science workflows.

Hello, CodeMirror 5! by @mootari

Demo Stuff

SQLite by @observablehq

Observable has built-in support for SQLite, “a small, fast, self-contained, high-reliability, full-featured, SQL database engine” and “the most used database engine in the world.” Observable’s SQLite client uses sql.js, an Emscripten port of SQLite. After attaching a SQLite file to your notebook, create a SQLite database client by calling file.sqlite(). Here we’ll explore the Chinook sample database which represents a music library. You can also call SQLiteDatabaseClient.

localStorageView: Non-invasive local persistance by @tomlarkworthy

Lets make it simple to add local storage to a UI control (e.g. @observablehq/inputs) We exploit back-writability and input binding to avoid having to mess with existing UI control code. localStorageView(key) creates a read/write view of a safe-local-storage. Because it's a view it can be synchronized to any control we want to provide persistence for. We avoid having to write any setItem/getItem imperative wiring. If you want all users to share a networked value, consider shareview.

Hacking Observable's new Cell Modes to use SQL by @asg017

2021-12-01 Update: Rejoice! Observable now has SQL cells as a proper feature.

UI, UX and Web Design

Observble notebooks promote resuse. A number of useful components were developed which can be used in other Dataviz notebooks. Mathieu Jouhet wrote a stunning library + tutorial notebook on Rounding polygon corners in SVG

That colorful font by @kelleyvanevert

That colorful font from the browser extension, What have you made today? This is the original font face, I think: Type With Pride

D3 Charts by @d3

A guide to D3’s reusable example charts D3 Charts help you get started quickly with D3. To use these charts in your application, copy-paste or import the chart function into your code, then pass the chart function your data and any options. The chart function returns an SVG element that you can then insert into the page. The chart functions are designed to work out of the box: the options are all, well… optional, with reasonable defaults.

Rounding polygon corners by @daformat

In my previous notebook about how to find the union polygon of overlapping rectangles I've included a polygon corner rounding algorithm as a post-processing step but didn't explain how it worked.

Stencil all the things! Distributing circles around a shape by @daformat

Inspired by a tweet form Chris Gannon, I decided to explore this stencil effect by recreating it in a generative way. Instead of manually placing and colorizing each circle, I wanted to be able to dynamically generate them given any svg path. In this notebook we'll learn how use the svg APIs, sprinkled with some vector math, to randomly distribute shapes along any given path. Let's stencil all the things! Learning how to fish Now, in the end, this effect is pretty simple.

Upset Plots with Observable Plot by @spren9er

Venn diagrams can be used for comparing two or three sets. They give you an overview of intersection(s) of sets. However, when you have more than three sets to compare, Venn diagrams are not appropriate anymore. One way of visualizing intersections of multiple sets is an Upset Plot. We don't go into detail of how you read Upset Plots here, you can take a look at the official website or paper. There are libraries for R (UpSetR) and JavaScript (UpSet.js) available for creating your own Upset Plots.

Observable TimeChart by @observablehq

TimeChart is an importable component for visualizing time-series data. It can show many signals time-aligned as space-efficient horizon charts. If you hover over a chart, all charts on the page show the value at that time, aiding interpretation. To use in your notebook: Then, call TimeChart and TimeAxis to create charts as described below. Configuration TimeChart expects a consistent x-axis across plots so that you can see coincident patterns.

Interactive BCG Matrix Component by @analyzer2004

The BCG Matrix (aka Growth-share matrix) provides a strategic view for analyzing business units, products or services by splitting them into four quadrants according to the growth and relative market share: question marks, stars, dogs and cows. Each quadrant has its own set of unique characteristics that represents a certain degree of profitability. This helps companies decide where to focus their resources to generate the most value, as well as where to cut their losses.

Control Panel for Observable (Now with better responsiveness) by @analyzer2004

Control Panel is a set of components to assist in easily building a compact and neat argument UI for visualizations. This is extremely useful when the chart has a large number of arguments. It allows the user to adjust all the arguments while simultaneously seeing the changes without the need to scroll back and forth.

Composing viewofs with the view literal by @tomlarkworthy

Lets make custom UIs on Observable easy by composing views. We wrap the amazing hypertext literal with a interceptor that looks for key, view arguments. It uses the key to determine what field to map the view's value to in the container. The syntax of a 2 element array is inspired by Object.entries(...). By reusing the hypertext literal you are able to build your custom ui view using HTML, the best abstraction for layout.

CSS Variables #1: Basics by @gianordoli

CSS variables is a native way of creating ... variables in CSS! Their official name is actually “CSS custom properties” and they have been supported by all modern browsers since at least 2017. Unlike preprocessors variables, CSS custom properties have access to the DOM. A consequence of that, and the greatest advantage of them over preprocessor variables, is the possibility of declaring values that change based on media queries. More on that later.

Color Input / Observable Inputs by @observablehq

This input specifies an RGB color as a hexadecimal string #rrggbb. The initial value defaults to black (#000000) and can be specified with the value option. The color input is currently strict in regards to input: it does not accept any CSS color string. If you’d like greater flexibility, consider using D3 to parse colors and format them as hexadecimal. If you specify the datalist option as an array of hexadecimal color strings, the color picker will show this set of colors for convenient picking.

Form Input / Observable Inputs by @observablehq

The form input combines a number of inputs into a single compound input. It’s intended for a more compact display of closely-related inputs, say for a color’s red, green, and blue channels. You can pass either an array of inputs to Inputs.form, as shown above, or a simple object with enumerable properties whose values are inputs. In the latter case, the value of the form input is an object with the same structure whose values are the respective input’s value.

Observable Plot: Box by @observablehq

This notebook demonstrates how to use composite marks (arrays of marks) and transforms to create a box plot using Observable Plot. These functions (boxX and boxY) define a composite mark suitable for a box plot, applying the necessary
statistical transforms. The boxes are grouped by y or x, respectively, if present. This map function returns only outliers, returning NaN for non-outliers (based on the whisker definitions below).

Hello, citation-js! by @fheyen

github.com/larsgw/citation.js Paste one or more DOIs, or other identifiers into the input textbox and this notebook will display the citation(s) as text and bibtex.

Tree Input by @tophtucker

Observable has a lovely default tree inspector for hierarchical data: But you can’t select a child and refer to it in code. With this you can: Options: children: accessor; defaults to d => d.children name: function returning string or DOM node; defaults to `d => d.

Responsive grid by @tomlarkworthy

Place views on responsive grid panels to create mobile friendly composite views quickly. Simple Uniform Coordinates... yet responsive The controls above are for building intuition over the coordinate system and grid parameters. Play with it! (you can instantiate it against your own grids too) Formally, there are n panels which rearrange to suit the screen resolutions. Typically three panels per row for desktop and 1 per row on mobile.

Radar Chart by @saneef

Learning D3.js References Radar chart by Ben Welsh West Coast weather from Seattle to San Diego by Eric Lo Brand Personality Radar Chart by Jacob T.

Tooltip (D3 Convention) by @clhenrick

This version of a Tooltip uses the D3JS convention of `selection.call()` to register the Tooltip's behavior. This is an improvement on my original version of a Tooltip Component.

How to draw an arrow between 2 dots? by @neocartocnrs

undefined

Simple pencil + ink pen effect for SVG path using filters by @oliviafvane

"By Quentin Blake" This is a simple way to do a pencil or ink pen effect for an SVG `<path>\, using the \<feTurbulence/>\` noise filter. For more pencil effect examples on Observable, have a look at @matthewturk/pencil-effect-filters, and which also simplify and curve the path @coreygirard/pencil-mercator, @d3/sketchy-earth and @visionscarto/pencil-globe.

Tutorials

The Observable community likes to help others learn. There are tutorial notebooks dedicated to a wide variaty of topics. I write a lot about how to build applications on Observable, and a focus area for me in 2021 was scaling UI development.

Observable pour les géographes by @neocartocnrs

Depuis plus d'un an, je réalise des cartes et des visualisations de données avec Observable. Dans ce carnet, j'essaie d'expliquer ce que j'ai compris et pourquoi je considère que cet environnement est idéal pour créer des cartes. Je vais probablement améliorer ce matériel de présentation au fil du temps. N'hésitez pas à commenter et à suggérer des ajouts/changements. Commençons par une petite présentation introductive (Thanks Reveal.js). Ressources https://towardsdatascience.

Analyzing Time Series Data by @observablehq

Approaches for working with data that changes over time Many of us work with data that is about time, tracking fluctuations by hour, day, week, month, etc. We need to know what caused a spike or dip, learn why the actuals diverged from the forecast, characterize typical daily, weekly, and seasonal patterns, or see how the correlation between two metrics depends on time.

Scaling User Interface Development by @tomlarkworthy

Backend development is rapid, and yet... custom user interface development is still a huge timesink. UI development is a project bottleneck. This living document is a growing list of techniques I have learnt trying to overcome the UI development bottleneck. When I create complex applications, I need contextually sensitive user interfaces.

Union polygon and area of rectangles (line sweep algorithm) by @daformat

Given a set of unrotated and potentially overlapping rectangles, we want to compute the total area covered by their union and get their union polygon(s). One way to proceed is to first use a line sweep algorithm in order to subdivide the surface of the rectangles to a set of non-overlaping rectangles covering the same area. In other words, we want the resulting rectangles to be either contiguous or completely separated.

How to make a nice scale bar by @jgaffuri

Here are some hints to make a scale bar with nice rounded values and divisions, for a map. Usually, a scale bar has to fit within a rectangular component, with limited dimensions. It is unlikely that the scale bar length corresponds to a nice rounded value (such as 100km) which can be easily divided into other nice values (such as 25km, 50km, 75km). In such case, it is more suitable to shorten a bit the scale bar to a lower length value which makes it nicer (see here for example).

Plot for mathematicians by @mcmcclur

Observable Plot is a free, open-source library from the folks at Observable designed to make it easy to generate many common types of charts for the visualization of data. The documentation is extensive and generally excellent. It's (understandably) focused on its usage with data, though, which obscures how easily it can be used to generate nice mathematical plots. The purpose of this notebook is to illustrate how we might use it to generate simple graphs of functions.

Tools

The Observble community has built some impressive tooling to support data practitioners. The biggest news of the year was the introduction of Plot by the Observable team, which allows rapid production of informative charts.

Observable Plot by @observablehq

Observable Plot is a free, open-source JavaScript library to help you quickly visualize tabular data. It has a concise and (hopefully) memorable API to foster fluency — and plenty of examples to learn from and copy-paste. In the spirit of show don’t tell, below is a scatterplot of the height and weight of Olympic athletes (sourced from Matt Riggott), constructed using a dot mark. We assign columns of data (such as weight) to visual properties (such as the dot’s x), and Plot infers the rest.

Data Wrangler by @observablehq

Data wrangling is often a complex and time consuming part of quantitative work.
This (experimental) Wrangler function provides an interface to guide code
composition and also display the results. It uses the
Arquero package, a
robust data wrangling library based on the grammar of data manipulation. To get started, import the Wrangler function from this notebook: Then, pass your data into the Wrangler function, and access the results: As an example, here is an exploration of U.S. protest data
(source).

Introducing Observable Plot by @observablehq

We are thrilled to announce Observable Plot, a new open-source JavaScript library for exploratory data visualization. Observable’s mission is to help everyone make sense of the world with data. To succeed, we need to make visualization easier and faster, both to learn and to practice. Less chore, more joy. We believe people will be more successful finding and communicating insights if they can “use vision to think” instead of wrestling with the intricacies of programming.

Summary Table by @observablehq

Want a quick visual overview of a dataset? To get started, import the new (experimental) SummaryTable function from this notebook: Then, pass your data into the SummaryTable function. As an example, here is a summary table of the Penguins dataset (described in detail here).

Future of Data Work: Collaboration and No Limits by @observablehq

Imagine all the ways that data flows through an organization. People explore and analyze data. People ask critical questions of data. All of these people bring diverse skills when engaging with data - data scientists, data analysts, data engineers, developers, product managers and decision makers. All of these people seek insights from data for decisions toward better business outcomes, but they all use a different set of techniques and tools to do their jobs.

Introducing Dataflow, a self-hosted Observable Notebook Editor by @asg017

Dataflow is a new tool that lets you run, edit, and compile Observable notebooks locally on your own computer, with any text editor you want! If you have ever wanted to save + run Observable notebooks as files on your computer (for easier integration with git), then Dataflow will be very helpful! Quickstart Check out the documentation for more! In Dataflow, Observable notebooks are files on your computer, with a `.

Plot Tooltip by @mkfreeman

Two approaches for easily adding tooltips to your plots Tooltips on all plots Want to add tooltips to all of your plots? Import plot from this notebook, then simply set a title attribute for any mark in your plot call (note, this apporach only works with Plot.plot() calls, not `Plot.MARK().

Future of Data Work: Q&A with Mike Bostock by @observablehq

Mike Bostock has been busy. Our CTO and co-founder is the author of D3.js, an open-source library for data visualization which celebrated its 10th anniversary this year, and Observable Plot, a new library for exploratory visualization that builds on D3. Not to mention designing and developing features for the Observable platform. For this week’s Future of Data Work series, I sat down with Mike for a Q&A about his motivations and how he hopes people work with data in the future.

What’s new in Observable Plot (0.2.7)? by @observablehq

For more on the latest releases, please see our Changelog. Plot now allows you to inspect scales! Rendered plots (either SVG or figure elements) now expose a plot.scale(name) method that returns an object describing the scale in-use, including any materialized defaults. For example, say you made a scatterplot showing the rise of global surface temperature anomaly using a quantile color scale. To retrieve the color scale for this plot, call plot.scale passing in the name “color”.

Building a Better Beeswarm by @yurivish

Beeswarm plots are a fun, visually pleasing alternative to histograms that can be used when data is sufficiently small and you want to plot every point. This notebook contains an algorithm for accurate beeswarm plots with circles that vary in radius, using a simple implementation of this algorithm by James Trimble. In addition to being faster to compute than a force-directed layout, it has the advantage of improved accuracy since it preserves the precise x position of each data point.

G2Plot 2.3.23 Released 🎉 by @me-momo

G2Plot is an interactive and responsive charting library based on the grammar of graphics, which enables users to generate high-quality statistical charts through a few lines of code easily. G2 in "G2Plot" means the grammar of graphics and pays homage to ggplot2. What's New in G2Plot 2.3.23 We've added two new plot types: Facet and Violin, support localization, enhance several plots, like Sunburst, Stock, Heatmap and Box and so on.

Observable + GitHub 💞 by @observablehq

Visualizing data is more important than ever There are ~56M+ developers on GitHub and growing. As more developers collaborate, complexity increases. It’s easy for technical matters to consume all our attention, but are we asking the right questions? Is this the right data to be looking at? What are we trying to understand? How to we stay focused on insight? On thinking? How do we - all of us - shift our behavior in order to find answers together? Visualizing data is ONE of the ways we find answers together.

Starting Github Action Workflows From Observable

Tom Larkworthy — Mon, 27 Dec 2021 09:06:51 +0000

Using Observable as the front end for Github Actions is a great way to provide a nice interface without the hassle of hosting anything. Recently I developed a notebook library which allows authenticated and unauthenticated users to kick off Action workflow via a repository dispatch.

A repository dispatch triggers Github Action workflows via an authenticated HTTP request. See the documentation on Github. From there you can do all kinds of things, even programmatically creating commits.

We document two major ways to trigger actions from an Observable notebook.

authenticated, where notebook readers provide the access token to trigger a workflow
pre-authenticated, where the notebook author configures a proxy using their own creds, so anybody can trigger workflows on the author's blessed path.

To use

import {dispatch, createDispatchProxy} from '@tomlarkworthy/repository-dispatch'

Repository Dispatch Client Library for Observable

The authenticated dispatch function takes an access token, plus the other options, to perform the dispatch

async function dispatch(
  token,
  { owner, repo, event_type = "event_type", client_payload = undefined } = {}
)

Example Usecase 1: Pass Personal Token In Notebook

To use our simple dispatch function, we need a Github API access token. A simple way is to ask the reader for one.

Note: If you use localStorageView you can remember the input across page sessions in local storage.

There are situations where you don't want the reader to provide a token. For this, we need to introduce a secure environment.

Example Usecase 2: Use Shared Secret Stored in Webcode.run to pre-authorize a configuration

Code executed on webcode.run is remote, and so secrets are not exposed to notebook readers. By creating a proxy to do the request, we can expose Github workflows to the public without requiring them to supply tokens. Instead, the secret can be set in the inline UI (or at @endpointservices/secrets). But default, the secret should be called github_token, but you can override this in the createDispatchProxy arguments.

which you can call to invoke a github workflow

Note, configuration of the dispatch function has to be done when creating the proxy so that the endpoint cannot be used against arbitrary repositories. An opt-in exception is that when the client_payload arg is set to undefined, the client can pass a value of client_payload as the first argument (this is to enable passing data into a workflow).

viewof remoteDispatch = createDispatchProxy({
  owner: "tomlarkworthy",
  repo: "octokit-test"
})

The result of a createDispatchProxy call is a view. The UI is the default webcode.run UI, which provides a method of setting the secret once you are logged in. You will need to create a stored secret called github_token and bind it to the endpoint. If you forget to configure the secret, the underlying Octokit API will throw HTTPError: Not Found exceptions. Your notebook needs to be published for it to work when you are logged out.

The data channel of the returned createDispatchProxy view is a dispatch function. When called, it will run the dispatch remotely, but importantly, you do not need to pass any authentication credentials. So you can just place it in a button for anybody to use. Try mine below.

Refer to the source notebook for the interactive demo

The above workflow will trigger this workflow which in turn write back into the repository as a log entry found here. So you can see for yourself that your triggers are registered by inspecting the timestamps in the logs!

The example here will cause a commit to occur in a repository you would not normally have permissions to! Now imagine what is possible when you can offload storage into git repositories for public users on Observablehq!

Writing workflows

I recommend Triggering GitHub Actions Using Repository Dispatches for a good overview of the technique.

At the beginning of a workflow file (stored in /.github/workflows/<name>.yml) you announce how the workflow is triggered. For an externally HTTP triggered workflow you use repository_dispatch. You also can optionally restrict what the event_type must be through the types array property:-

name: HTTP triggered workflow
on:
  repository_dispatch:
  types: [start-example-workflow]

Next, you list jobs with steps. Steps can include anything off the vast Github Actions marketplace or just simple shell scripting!

jobs:
  commit-timestamp:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Log workflow execution in log
        run: |
          mkdir -p ./log
          echo "$(date '+%Y-%m-%d %H:%M:%S') inbound HTTP" >> ./log/http.log
      - name: Commit updated log
        run: |
          git config --global user.name 'Log External HTTP'
          git config --global user.email 'robot@webcode.run'
          git add ./log/http.log
          git commit -am "Log inbound HTTP"
          git push

Observable + Github Actions

Observable is a great way to provide a lightweight interactive UI with minimal development effort. When paired with Github actions you can use Observable to create a nice GUI over the Action primitives. You can also use Actions to enhance the Observable experience, for example, auto-backing up notebooks into a Github repo using the onversion hook.

Creating long lasting content on Observable

Tom Larkworthy — Tue, 07 Dec 2021 10:10:55 +0000

I am 100% convinced Observable is the right way to share computational thought. Sometimes though, it feels like notebooks are only read once during their birth on the /trending page. However, I am happy to report you can create long-lasting content with lots of repeat visitors and it is surprisingly straightforward to do. In this article, I will share what I have learned on my Observable journey.

I took data-driven approach to find out what works. After wiring my pages to Plausible Analytics I could finally get a birds-eye view of my readership. I found the results fascinating! Here are my page views in the last month (Nov. 2021):-

Collaborate and solve a platform need

My #1 content by a long shot is Composing viewofs with the view literal. This is a super useful utility that helps create complex UIs on the Observablehq platform. It fills a gap in the platform primitives, and clearly, I was not the only one who needed it. I collaborated with @mootari and @mbostock on it, and it was developed to support a consulting job that needed a production-level UI.

Now, I have written many libraries for Observable. None of them do as well as this one. I think this one resonated so much harder because it was the product of a business need, and it was refined via conversations with others. So my first tip for creating great content is: talk to others in the community! Use the forum or leave comments on people's notebooks!

However, most of my other utilities are not in my top ten, which implies that it is actually very hard to develop a library that others will use. It took me 6 months of full-time work on Observable to understand enough to craft that library. So generally, I would say it is not a very easy path. The saas-tutorial and Firebase are other examples of Observable utilities that have done quite well, but again, they took a lot of effort to produce.

Go viral!

My next most popular content was created well over a year ago and is still going strong. The Most favorited Hacker News posts of all time went stratospheric on Hackernews, getting 1260 upvotes and linked to all over the web.

It is very satisfying to me that good content like that is rewarded with a steady stream of traffic continuously for years. It gives me the motivation to keep trying. The Hackenews Favourites notebook really is an excellent article as it surfaces the crème de la crème of Hackernews which was previously unknown.

Obviously writing good viral content is kind of hard. If I had the formula to do it I would have more, but I am satisfied that if you do figure one out, you will get good readership numbers. The notebook Sign a PDF and Adobe: Go Fuck Yourself also was a Hackernews hit, though not to the same level, but that's also why it has reached the top ten.

Promote your content outside of Observable

The notebooks mentioned in the previous tips represent my "smash hits", which are a bit random to produce. They were generated when inspiration hit. On the other hand, the remaining notebooks are doing well, and did well for fairly simple and repeatable reasons. This section is the most important one as this is where luck no longer plays a role.

notebook	views	likes
switch-dataflow	126	9
merge-dataflow	80	9
mip	62	27
sound-cloud-reactive-audio-visualizer	37	24

What's interesting with these notebooks is that their success is not dependant on Observable "likes". The real reason why these set of notebooks do so well is that they were all promoted outside of Observable.

The switch and merge dataflow tutorials are blog posts that were syndicated to dev.to and medium using the meta-notebook Blogify. The Mixed Integer Programming (mip) notebook was shared on r/optimization and the reactive visualizer on r/edmproduction.

These notebooks were not big hits! But this is what is really cool about them. By simply cross-posting them off the platform has a huge effect on their longevity, regardless of their popularity on Observable. Their SEO is good. This is great news. Performance on /trending is not indicative of performance elsewhere. You simply have to cross-post.

Conclusion

I am not a gifted content producer, but you don't need to be! It's easy to share your work on Reddit, and that act alone makes a huge difference to the longevity of your work. I think most people on Observable produce much better work than me. You are not doing it justice if it stays only on the platform. If you just go out and share it on places like r/dataisbeautiful or hackernews you will amplify your impact massively. This is the simple secret of creating long-lasting content on Observable.

How to 1-of-n switch Dataflow streams on Observable

Tom Larkworthy — Wed, 24 Nov 2021 12:14:59 +0000

In this series, I will explore programming techniques for the notebook platform Observable. Today I am looking at how to direct a reactive Dataflow stream to one-of-n downstream cells. This article assumes you are familiar with Observable's non-linear reactive program flow already.

Sometimes we want control the dataflow with logic. If we have a cell called data streaming values, we might like to direct those updates to either cell_1 or cell_2.

You can programatically trigger dataflow by calling dispatchEvent on its enclosing view.

You can create a lightweight view with Inputs.input

Now dataflow is guided by the value of the choice variable.

Give it a try on the live notebook at @tomlarkworthy/switch-dataflow

I hope you find that useful and that you find other useful ways to manipulate dataflow!

Productionizing Observable Notebooks with 3rd party Active Monitoring

Tom Larkworthy — Sun, 14 Nov 2021 11:43:13 +0000

Productionizing Observable Notebooks with 3rd party Active Monitoring

I keep my 200+ public notebooks on the observablehq.com platform running by actively monitoring them with a 3rd party monitoring tool uptimerobot. Here is how I got it connected, and how I did it unobtrusively without altering the original notebooks.

Broken notebooks are a big problem. I have encountered many broken notebooks on observablehq.com, and then much later, I have written many more. The thing is, I did not know my old notebooks were broken until I randomly checked! I don't want to host broken notebooks, but I can't be checking 200 notebooks every week, it doesn't scale! Instead, I figured out a way to bring off-the-shelf monitoring solutions to the notebook ecosystem. The best thing is that it can be applied to any notebook. There is no special library or anything to depend upon etc.

Using the Observablehq runtime to create a Healthcheck Metabook

Did you know that all notebooks are packed as ES6 modules, and you can run them inside the open-source Observablehq runtime programmatically? (see Advanced Embedding and Downloading) This means a notebook can find and run code found in another notebook. I call these types of notebooks metabooks as they are higher-order notebooks (notebooks whose input is another notebook).

So, the healthcheck metabook, given a target notebook, runs the target on an internal Observablehq runtime and looks for errors.

  const [{ Runtime }, { default: targetNotebook }] = await Promise.all([
    import(
      "https://cdn.jsdelivr.net/npm/@observablehq/runtime@4/dist/runtime.js"
    ),
    import(`https://api.observablehq.com/${settings.target}.js?v=3`)
  ]);
  new Runtime().module(targetNotebook, (name) => {
    // cell observers
    return {
      ...
      rejected(error) {
        // cell threw an error
        ...
      }
    }
  }

Using the runtime hooks we can detect if a cell throws an error, which will pick up many common forms of notebook rot. To attach 3rd party monitoring though, this error detector needs to be exposed as a HTTP service for 3rd party consumption.

Exposing a HTTP service with WEBcode.run

We use a Observable native functions-as-a-service runtime WEBcode.run to expose an endpoint on the internet.

With WEBcode.run you define a handler for requests that write back to a response. Because it executes in the notebook, you get access to all the notebook functionality. In our case, we decode a target notebook from a URL parameter. The API is modelled after the Express 4.0 API (see req, res for details).

endpoint(
  "default", // For a simple URL we use the default name
  async (req, res) => {
    const target = req.query.target; // Read the target notebook.
    run(target, excludes); // start health checking

    setTimeout(() => {
      // There is no clear stopping point so we just run it for X seconds
      const errors = viewof errors.value; // collect errors
      res.status(errors.length > 0 ? 503 : 200).send(
        JSON.stringify({
          errors: errors.map((e) => ({
            cell: e.cell,
            message: e.error.message
          }))
        }, null,2 )
      );
    }, 5000);
  },
  {
    reusable: false, // This does not support concurrent operations
    modifiers: ["orchistrator"] // This endpoint can call other endpoints
  }
)

Our handler starts the healthcheck process, and after 5 seconds reports, if errors are found.

So, to check for errors on notebook @tomlarkworthy/view, we make a request using curl to

curl 'https://webcode.run/observablehq.com/@endpointservices/healthcheck?target=@tomlarkworthy/view'

Because 3rd party monitoring tools typically just check HTTP status codes, our handler returns a status code 503 (unavailable) if errors are detected, or 200 (OK) otherwise.

Attaching UptimeRobot

UptimeRobot is a very simple to use active monitoring service and it will provide 50 monitoring jobs for FREE!

Active monitoring will poll the provided URL on a schedule, log the status, and alert via email/SMS/webhook if an error code is seen.

Unit tests, Sentry

Being able to actively monitor my notebooks for errors has completely changed my anxiety levels. By looking at my dashboard I can see that all my main notebooks are running well. As I am building a business upon Observablehq, I must keep everything working, so seeing my code has run successfully is reassuring.

The biggest change is that when an error is reported by an end-user, I now convert that issue into a unit test. Because unit tests errors are runtime errors, a healthcheck over a notebook containing a test suite will alert on unit test failures. So now I can fairly confidently ensure that particular issues never happen again.

I also use sentry.io to automatically report errors, whether generated by user usage, or by healthcheck monitoring (see this notebook). Again, for low usage, Sentry.io is a FREE service. Sentry compliments UptimeRobot by logging context around an error, and also it runs in user devices like iPhone which can sometimes have their unique issues.

I am very excited that we can take a developer-friendly product like Observablehq and still fit it into the existing DevOps infrastructure. It suggests that building high-quality software on Observable is possible, and maybe preferable. I love Observable, in that, for all its workflow cleverness, it's still just Javascript running idiomatically in a browser, and thus, we can leverage existing specialist tools that already exist in the ecosystem. There is no need to reinvent the wheel when using Observable! I hope this story helps you get to the reliability levels you want out of your software hosted on Observablehq.

I tweet lots of Observable stuff at @tomlarkworthy, and occasionally launch things on Product Hunt

How to dataflow merge on Observable

Tom Larkworthy — Fri, 05 Nov 2021 19:34:44 +0000

How to dataflow merge on Observable

In this series I will explore programming techniques for the notebook platform Observable. Today I am looking at how to merge multiple reactive Dataflow streams into a single stream. This article assumes you are familiar with Observable's non-linear reactive program flow already.

A common situation is this: you have a notebook that performs some useful task, and you want to offer a few different ways to start that task. For example, starting something based on configuration obtained from either URL parameters, local storage or a manual UI. It's a little trickier than you would hope because, if each of these methods are in their own cell [1], then we have multiple distinct data flow pathways converging into a single flow.

[1] and they probably should be because of separation-of-concerns

The core difficulty is if there is no data in a source, its dataflow won't tick, so if the business logic depends on all the input sources, the business logic will not tick either. So this pattern described here will joins the business logic to the sources a different way so that if any of them tick, the business logic will tick too.

In ReactiveX terms Observable's dataflow is a combineLatest across cell streams, but we want a merge.

So here is how you can do it with views and Inputs.bind.

Step 1. Create an unresolved view for the business logic to depend on

We can quickly create an unresolved view with Inputs.input(). We will fan-in sources to this view.

If the business logic extracts parameters from the config variable, then it will not run until it is set to a non-undefined value (note grey bar on the left of the cell after notebook startup).

Our example business logic will flash the notebook in a color set by the config.

2. Setup all the sources to be views

Next we need the sources to be emitting their config to a view's value.

Often, a source is already a view:

But even if it isn't, it is trivial to create a view from a dataflow variable by writing into an Inputs.input()

3. Bind the source views to the fan-in view backwards

Now, if either of the sources fire, the business logic is run. Give it a try on the live notebook on at @tomlarkworthy/merge-dataflow

I hope you find that useful and that you find other useful ways to manipulate dataflow!

The Fully Encapsulated Monitoring Notebook

Tom Larkworthy — Sat, 23 Oct 2021 11:55:23 +0000

Imagine, in a page of code, hosted entirely on the web, with zero tools to install or infra to manage, you could have a state-of-art programmable data visualization system recording anything reachable on the internet. Bold claim? I have built two such systems and I am so excited by the possibilities I wish to excitedly share the architecture.

IDE, Documentation, Code Host and Front End: Observablehq.com

Observablehq.com is a mind-blowingly simpler platform that can replace: Github, CI, VScode, static hosting, Google docs, and containers. It hosts, runs, and provides an IDE for client-side Javascript. Furthermore, it provides a novel reactive notebook format, so code can be written non-linearly and the program output is interleaved with the code. It’s also a literate programming environment, so notebooks also support documentation, multiplayer commenting, and collaboration.

Observable is a single tool that can replace numerous critical tools required in a software project. Including development, technical documentation, and hosting.

Charting: @observablehq/plot

Observablehq.com, is the hottest place for Dataviz, so it naturally hosts a number of state-of-the-art tools such as plot. Plot stands out by being so fast to develop with, but Vega-lite is another great option. Check out some of the workshops on how to use these tools effectively.

(Plot) Analyzing Time Series Data in Observable: Hands-on Workshop
(vega-lite) Observable: Vega-Lite: A Crash Course

To do queries on the client side I would use Arquero for data operations. There is an interactive intro to Arquero through Data Wrangler.

UI: @observablehq/inputs

For simple controls, Observable has an excellent selection of off-the-shelf components through @observablehq/inputs. With some work, you can make complex UIs too (see Scaling User Interface Development).

Realtime Database: firebase.google.com

As Observable is a reactive client-side environment, it works really well with reactive client-side databases like Firebase. When I am choosing for monitoring systems, I look for: cheap writes and low latency. So I tend to pick the Firebase Realtime Database over Firestore for monitoring applications. (FWIW I pick Firestore for online transaction processing applications).

API Endpoints: WEBCode.run

Monitoring solutions often need APIs, so they can respond to external services (e.g. queries, datapoint ingestion, OAuth handshakes). WEBCode.run extends Observable with an SDK that allows notebooks to host serverless endpoints. These endpoints can utilize secrets and scheduled functions so are a great fit when you need to interface with external systems and it can be all done without leaving observablehq.com.

Example Monitoring Notebooks

I have built two notebooks so far that are self-enclosed systems, and they have both provided immense value to me. I intend to do many more.

1. Realtime Inbound Request Log

The Realtime Inbound Request Log notebook hosts an endpoint and puts all inbound requests in a log. When you view the notebook the log entries are queried and some summaries are provided. This notebook was developed to help me sniff request headers on a remote machine not under my control, by scripting the remote machine to query the instrumented endpoint.

2. Latency Monitor

The Latency Monitor notebook is a full end-to-end prober solution. It periodically schedules traffic to a system, measures the latency, and summarizes the results in a dashboard.

This has been in operation for nearly a1 year and helped quantify latency performance.

Everything in one end-to-end environment.

I am delighted that everything is in a single artifact that can be shared with a URL. This is not some low code compromise either, all the parts work together providing the end-user with a state-of-the-art experience. Realtime data dashboards using state-of-the-art DataViz libraries, stitched together with a modern language using tools that support version control, step debuggers, etc.

By having the tools to modify the monitoring pipeline next to the technical documentation and realtime output of the production data allows anyone to maintain it with no context switching.

Overall it is a joyful development experience and operationally simple.

Credits

Article hosted on Observable and syndicated with Blogify
Photo by Myriam Jessier on Unsplash

Securing a Serverless Multi-Tenancy Puppeteer Service

Tom Larkworthy — Thu, 10 Jun 2021 08:15:59 +0000

If you ever try to stand up a Puppeteer service you will almost immediately find it is difficult to secure when running inside a Docker environment.

I love my serverless, so I was not prepared to take no for an answer. And with a lot of sweat, I think I able to stand up a Puppeteer service with full customer isolation and protection again serverside scripting from within a multi-tenancy docker container.

Customer Isolation

Customers should not be able to view each other's data.

--no-sandbox

Chrome itself is natively very good at sandboxing tabs for security reasons. Ideally, we would simply just exploit the inbuilt security model, e.g. put each customer in their own tab(s). Not so fast though! Unfortunately, Chrome won't boot under that configuration. The way that sandbox is implemented does not work containers, as a result, nearly every Dockerfile for Puppeteer on the internet launches with the --no-sandbox flag.

--cap-add=SYS_ADMIN

The few Dockerfiles I could find without --no-sandbox have added the SYS_ADMIN security capability. This is one solution to keep the sandbox, but most managed docker environments don't expose this control, unfortunately. So I needed a different way to work on Serverless.

Linux process isolation

Normal Linux processes cannot mess with each other's memory. So the OS approach for customer isolation is to run a different browser for each customer.

Resource isolation

You still need to be careful though, as even separate Chrome processes can still access common resources (e.g. filesystem). In particular, user sessions, cookies, and website cached data need to be stored in different directories for each customer

--disk-cache-dir
--media-cache-dir

Protection against Serverside Request Forgery

A Puppeteer service essentially allows end-users to run code within our infrastructure. The big danger is that the Puppeteer instance will become a bastion for network intrusion. This is not an academic thing either, a ton of exploits observed in the wild used Puppeteer or similar to launch attacks via serverside request forgery attacks. Check out

In particular, an easy attack is to have Puppeteer run a webpage that probes for Cloud metadata servers, which can then be used to obtain credentials.

So, we must prevent Chrome from accessing certain URLs and local IP addresses.

You can find a list of IP addressed to block on owasp.org.

--proxy-server

Chrome can be configured to use an outbound HTTP proxy server, which we can use to intercept and filter traffic. For our service, we used TinyProxy as it has a very low resource overhead (2MB).

The TinyProxy configuration then protects against access to sensitive IP addresses and domains.

Our translation to TinyProxy configuration can be found on Github

Exposing the Chrome DevTools protocol port

The most exciting thing for me is allowing users to script Chrome from within their browser environment remotely. This is enabled by exposing the Devtools debug WebSocket protocol.

To allow us to meter access, we expose a WebSocket endpoint that requires an access_token to be in the path. We can then verify the access_token, boot Puppeteer, and then proxy the public WebSocket endpoint to the internal WebSocket endpoint on demand.

Try it in your Browser without installation

With that in place, we are now able to offer Puppeteer access from right within a browser. Check it out, we are hosting it from within @observablehq in a notebook.

Full source code is available on Github and it is designed to be run on Google Cloud Run.

Firestore's Technical advantages

Tom Larkworthy — Tue, 11 May 2021 07:42:44 +0000

I have read some pretty poor articles bashing Firestore recently. Generally they completely miss the features set or cargo cult Postgres SQL. This article attempt to highlight the features of Firestore that you won't see with a Postgres solution (note I love Postgres), highlighting several area's where Firestore is the world #1.

Clientside first

It's designed for a direct connection to a mobile/webapp. This means it has a number of features that are unmatched in the market.

Latency compensation

Firestore maintains a local cache, so local writes and observable immediately. Greatly simplifying controller design. It even broadcasts writes to adjacent tabs in the browser for 0 latency across browser tabs. You ain't got time to implement that!

Offline persistence

The cache is backed by persistent storage, so your app works offline without much work. This is a huge feature that is difficult to get right and essential for a good mobile experience.

Authorisation rules

The database has a layer of security rules which are very flexible and can depend on values in the database.

Causal Consistency

Client SDKs observe their own writes first, and remote writes sometime later. Firestore guarantees that remote writes preserve their order. This is better than eventual consistency, its causal consistency and the best you can manage in a distributed setting.
The fact write order is preserved makes the system very intuitive, but many Firestore competitors do not guarantee this property which leads to weird bugs and second guessing.

1M concurrent clients

Its not so easy to support 1M concurrent connections, that's serious engineering work.

Spanner Backed

Firestore somewhat outclasses Postgres on underlying database technology too, being based on Google Spanner. Firestore is the most affordable way to access a Google Spanner based database.

99.999% SLA

Yes. You probably can't find a more reliable cross region database.

Multi-region yet strong consistency

Writes are replicated across multiple regions. This is one of the reasons why it is so reliable, it is resistant to single data centre losses. It can achieve this AND still be strongly consistent. It is simply not possible to configure Postgres to be multi region and be strongly consistent. This is really what Spanner brings to the table.

Transactions

Firestore can do atomic writes across documents, without caveats, without sharding. Very few distributed databases can achieve this in a multi-region setting.

array-contains-any joins

I have read that noSQL databases do not support joins at all. This is true for many NoSQL solutions, but not the full truth in Firestore's case. It is true query expressivity is lower than SQL.

However, thanks to the "array-contains-any" query you can retrieve a set of documents matching a set of ids in a single query. This is far more efficient than having to retrieve documents on the other side of a join one at a time. Thus a SQL with 3 joins can usually be performed with 3 queries in Firestore with the appropriate indexes. Though, to be fair Postgres has the upper hand here.

Scalability

Firestore is true serverless with essentially unbounded scalability thanks to its Spanner backend. It also scales to zero so you only pay for what you use, unlike Postgres which has a fixed provisioning cost and an associated performance ceiling.

Conclusion

Postgres is a great default choice for a startup. However, if your product is used on mobile or across the globe, you might find Firestore a better match due to its state-of-the-art backend and client SDKs.

Disclaimer: I used to work on the Firebase databases

How Cloud Run changes Cloud Architecture

Tom Larkworthy — Mon, 03 May 2021 10:57:42 +0000

Cloud Run is interesting, it's a general-purpose elastic container hosting service, a bit like Fargate or Azure-Containers but with a few critical differences.

Most interesting is that it scales to zero, and auto-scales horizontally, making it very cost-effective for low traffic jobs (e.g. overnight batch processes).

It also runs arbitrary docker containers and can serve requests concurrently, meaning that for modest traffic you don't usually need more than 1 instance running (and you save money).

Its flexibility comes at the cost of higher cold starts though. Take a look at our cold start latencies for an on-demand puppeteer service in a low traffic region:

We are seeing cold start latencies of around 10 seconds, to boot up a 400MB container and start Chrome. This was annoyingly slow.

Not all our regions were that slow though, in one of the busier regions we saw a bimodal latency graph:

suggesting that 2.5 seconds is booting up a puppeteer instance and serving the request, and 5-7 seconds is booting the container. For busier regions often a container is running so that's why sometimes the cold latencies are much lower. (for completeness a warm latency measurement is 1.5 seconds, so probably 1 second is booting chrome, and 1.5 seconds is serving the request).

So... how could we speed things up? 5-7 seconds is spent on container startup. It's our biggest spender of the latency budget, so that's what we should concentrate on reducing.

One solution is to run a dedicated VM, though that loses the horizontal elasticity. Even so, let's do the numbers.

A 2 vCPU 2GB RAM machine (e2-highcpu-2) is $36.11 per month

Now Cloud Run has a relatively new feature called min-instances.

This keeps some containers IDLE but with no CPU budget, so they can be flipped on quicker. IDLE instances are still charged, BUT, at around a 10x reduced cost. The cost for an IDLE 2 vCPU 2GB RAM Cloud Run is $26.28 per month.

This gets pretty close to having your cake and eating it. You get lower latency like a dedicated machine, but also still horizontally elastic. It may even cost less.

For our application, we tried a min-instance of 1 and this was the result.

Our cold start latencies from container startup are decimated! We have not had to change any code.

I think this min-instances feature is a game-changer for cloud architecture. You can now get the benefits of dedicated VMs at a comparable price to dedicated VMs but with elasticity and image-based deployments. The new min-instances feature broadens the range of applications that serverless compute can address.

Our latency monitoring infrastructure and data is public.