DEV Community: Daniel Imfeld

Introduction to GeoJSON

Daniel Imfeld — Fri, 10 Feb 2023 00:00:00 +0000

Geographic data is ubiquitous in our lives. It is found in small applications like planning a vacation, and in complex worldwide topics such as population health and demographic studies. While the terms and systems involved may seem daunting at first, understanding geographic data can open up a lot of possibilities for visualizing and understanding the world.

Nearly every web application that works with geographic data uses represents it as GeoJSON. This is a short overview of GeoJSON and its structure to help make more sense of it.

In later articles in this series, I'll cover more interesting things you can do with geographic data and how to actually work with it, but this will provide a good foundation to build upon.

As the name implies, GeoJSON is just a format for representing geographic data as JSON. It defines six types of geometries, and a few different containers that group them together or provide extra information.

All images below are from geojson.io, a great tool for playing with and visualizing GeoJSON.

Points and Positions

The simplest geometry is the Point. Points are commonly used to indicate places of interest and other markers on maps where you just want to show the location, and don't need an area or boundary.

A Point contains just a single position. In GeoJSON, a "position" is an array with two numbers: the longitude and the latitude, in that order.

You may be accustomed to the more common "latitude first, longitude second" ordering. This is a common source of confusion when getting started, so it helps to think in terms of [x, y] coordinates instead.

A position array may have a third value to represent elevation, but this is usually omitted unless the application requires it.

Every GeoJSON geometry has a type field indicating what type of geometry it is, and a coordinates field that defines the shape itself. A point representing a random location in southern Australia looks like this:

{
  "type": "Point",
  "coordinates": [137.2, -32]
}

You may also place multiple points into a single geometry using the MultiPoint type. In this case, the coordinates are an array of positions.

{
  "type": "MultiPoint",
  "coordinates": [
    [137.2, -32],
    [133.1, -30]
  ]
}

MultiPoint should only be used when the set of points are conceivably actually part of the same object. There are better ways to represent multiple, independent points, which I'll cover later.

Lines

The LineString is useful when drawing lines on the map, such as when suggesting a route between two places. It contains a set of points that join together, connect-the-dots style, to form a line. As with MultiPoint, it contains an array of positions, but a valid LineString must contain at least two positions.

{
  "type": "LineString",
  "coordinates": [
    [-46.64, -23.577],
    [-43.2, -22.9],
    [-43.94, -19.89]
  ]
}

GeoJSON also provides the MultiLineString type, which represents a set of related lines. Its coordinates are an array of LineString coordinates, with each array member drawing a single line.

Although the lines are considered part of the same geometry, they are not connected together.

{
  "type": "MultiLineString",
  "coordinates": [
    [
      [-46.64, -23.577],
      [-43.2, -22.9],
      [-43.94, -19.89]
    ],
    [
      [-48.18, -21.77],
      [-47.46, -20.58],
      [-46.57, -21.78]
    ]
  ]
}

Making Geometries More Useful

A geometry on its own is just a shape, with no way to attach any other information to it. GeoJSON solves this with the Feature type, which wraps a geometry and allows you add any metadata you want.

{
  "type": "Feature",
  "id": "marker-001",
  "properties": {
    "name": "My favorite place",
    "icon": "star",
    "rating": "a billion"
  },
  "geometry": {
    "type": "LineString",
    "coordinates": [
      [-46.64, -23.577],
      [-43.2, -22.9],
      [-43.94, -19.89]
    ]
  }
}

The id property is optional but can be useful for linking a Feature back to a particular object in a database, or something similar.

There is no defined schema for properties — you can put any JSON object inside there.

Groups of Objects

A FeatureCollection allows you to group multiple Feature objects together in a single object.

{
  "type": "FeatureCollection",
  "features": [
    { 
      "type": "Feature", 
      "properties": { 
        "name":"my house"
      }, 
      "geometry": { 
        "type": "Point",
        "coordinates": [1, 2]
      } 
    },
    // And so on....
    { "type": "Feature", "properties": { }, "geometry": { } }
  ]
}

The GeometryCollection type is similar, but for raw geometries instead of Feature. You probably won't encounter GeometryCollection often. Usually, one of the "Multi" objects or a FeatureCollection is a better fit, but it's good to know about.

{
  "type": "GeometryCollection",
  "geometries": [
    { "type": "Point", "coordinates": [0, 51.5] },
    { 
      "type": "LineString",
      "coordinates": [
        [0, 51.5],
        [1.55, 50.7],
        [2.35, 48.85]
      ]
    }
  ]
}

Polygons

The Polygon is commonly used to draw boundaries around geographic regions or other areas. This is one of the most used geometries in GeoJSON, but they do add additional complexity.

Unlike a LineString, a Polygon must be explicitly closed. That is, the first and last points in the Polygon must be identical for it to be valid.

Furthermore, the points in a Polygon must be in counterclockwise order. This rule makes it easier for renderers to figure out which side of the polygon is the outside vs. the inside, without needing to do additional calculations.

There is no limit on the number of segments in a polygon. It can be as simple as a triangle or complex enough to follow the contours of a shoreline.

The last LineString above would be represented like this:

{ 
  "type": "Polygon",
  "coordinates": [
    [
      [0, 51.5],
      [2.35, 48.85],
      [1.55, 50.7],
      [0, 51.5]
    ]
  ]
}

You'll note that although there's just a single shape here, the array nesting looks more like a MultiLineString. This is because Polygon allows you to specify holes inside the shape as well.

The below Polygon is a square with two smaller square holes inside it. The first entry represents the outer ring that gives the overall shape, and the other entries define the holes inside the polygon.

{
  "type": "Polygon",
  "coordinates": [
    [
      [25, -23],
      [25, -28],
      [30, -28],
      [30, -23],
      [25, -23]
    ],
    [
      [26, -24],
      [27, -24],
      [27, -25],
      [26, -25],
      [26, -24]         
    ],
    [
      [28, -26],
      [29, -26],
      [29, -25],
      [28, -25],
      [28, -26]
    ]
  ]
}

While the outer ring of the Polygon has its points in counterclockwise order, the inner holes' points must be in clockwise order, which again helps renderers to determine which side of the edge is inside the polygon.

Ensuring that the points are in the correct order may seem difficult, but geometry libraries generally handle this for you. They also provide functions such as turf's rewind or PostGIS's ST_ForcePolygonCCW, which will take a polygon and "fix" the point order so that the outer line is counterclockwise and the inner ones are clockwise.

MultiPolygons

By now you can probably guess what a MultiPolygon is. These are very useful in representing geographic regions such as the state of Michigan or anything with islands, which can't be represented as a single contiguous shape.

It starts to get hard to read with all the nested arrays, but the coordinates for a MultiPolygon is just an array of Polygon coordinates. This example shows a MultiPolygon that contains two polygons — the square with holes from above, and a simple triangle.

{
  "type": "MultiPolygon",
  "coordinates": [
    [
      [
        [25, -23],
        [25, -28],
        [30, -28],
        [30, -23],
        [25, -23]
      ],
      [
        [26, -24],
        [27, -24],
        [27, -25],
        [26, -25],
        [26, -24]         
      ],
      [
        [28, -26],
        [29, -26],
        [29, -25],
        [28, -25],
        [28, -26]
      ]
    ],

    [
      [
        [28.5, -28.2],
        [25.1, -30],
        [30.2, -31.2],
        [28.5, -28.2]
      ]
    ]
  ]
}

Summing Up

That covers all the different types of objects that you can create in GeoJSON. In future articles in this series, we'll look more into actually processing data and working with these objects in real applications.

Avoiding Deadlock from Rayon Thread Pool Exhaustion

Daniel Imfeld — Mon, 09 Jan 2023 00:00:00 +0000

Rayon has a well-deserved reputation as one of the most useful Rust libraries. Its most famous feature is the ability to parallelize almost any iterator computation by simply calling par_iter() instead of iter(). This is a groundbreaking feature among programming languages in its class.

It transparently handles deciding how much to parallelize the iteration, spreading the data out to multiple threads, and then recombining everything at the end. It even maintains a pool of precreated threads, so that you can use par_iter liberally without having to worry about the overhead of creating and destroying threads.

But this thread pool did lead to an unexpected problem recently when I used Rayon in a data processing pipeline.

Pipeline Design

The pipeline code is part of a search project I’ve been working on. The project uses an article parser for HTML – think reader mode in your browser or Readwise – and the pipeline is designed to reprocess scanned data after installing an improved version of the parser.

There are four stages here:

Read the items from the database
Reprocess the content for each item, and send it to the next stage.
If an item’s processed output has changed with the new code, run it through the machine learning model to generate embeddings for semantic search.
Update the database with the new information.

Stage 2 uses Rayon to parallelize the processing algorithm across batches of content. I didn’t know it at the time, but stage 3 also uses Rayon when tokenizing the input to the ML model (which uses the excellentrust-bert crate). The flume crate provides channels to pass data from one stage to another.

But the pipeline would sometimes stall, with no obvious indication of problems other than a lack of progress and the CPU/GPU utilization dropping to zero.

Debugging

Since GPU support for Mac in libtorch is still encountering some minor issues here and there, I first suspected that might have been the problem, and so I resorted to the time-honored technique of sprinkling dbg! statements throughout the code to print to the console as things happened.

This finally narrowed it down to the model’s tokenizer starting to run, but never completing. But this part of the code was pure Rust and just did a bunch of string processing, with no torch code involved at all.

This seemed the least likely place for the pipeline to stall, until I looked into the tokenizer code itself and saw that it was using Rayon as well. Armed with this revelation and no other ideas of what could possibly be happening, I formed a theory.

The Problem

The ML model processes items in batches of 64 at a time, and running with an M1 Pro GPU this could take five to six seconds for an average batch. When the pipeline is working, it looks something like this:

Stage 2 sends a bunch of content to stage 3 through a channel.
Stage 3 reads the channel, and when it has accumulated 64 items that need to be run through the ML model, it processes them as a batch.
While the batch is processing, stage 2 continues to output content and fills up the channel again.
This repeats until all the items are processed.

Depending on the thread scheduling, it could be possible for stage 2 to fill up the channel again between the time when stage 3 would read items from the channel, and when it would actually start processing a batch.

So finally – and I should have just done this first – I loaded the program into the debugger, ran until it got stuck, and halted the program. As the debugger showed all the threads and their call stacks, I could see that stage 2 was running a bunch of Rayon threads, all waiting to output data into the full channel.

Stage 3’s threads were in some Rayon code, waiting for a free thread from Rayon’s thread pool to run on. But it couldn’t get one until stage 2’s code stopped blocking, and they couldn’t become unblocked until stage 3 processed its batch and consumed more data from the channel: a classic deadlock.

It turns out that the Rayon docs do actually warn about this too, albeit in a less commonly used function.

Fixing the Problem

I didn’t want to give up the parallelism in stage 2, because depending on how many of the items were different after reprocessing, it could actually be the bottleneck. Fortunately Rayon provides a simple solution by allowing manual creation of additional thread pools.

So stage 2 started out looking something like this:

// Simplified for readability
fn reprocess(
    db_items_rx: flume::Receiver<Vec<Item>>,
    processed_tx: flume::Sender<ScanItem>,
) -> Result<()> {
    for batch in db_items_rx {
        batch.into_par_iter().try_for_each(|mut item| {
            let processed_item = reprocess(item)?;
            processed_tx.send(processed_item)
            Ok(())
        })?;
    }

    Ok(())
}

And with the new thread pool, it changed to this:

fn reprocess(
    db_items_rx: flume::Receiver<Vec<Item>>,
    processed_tx: flume::Sender<ScanItem>,
) -> Result<()> {
    let pool = rayon::ThreadPoolBuilder::new().build()?;

    pool.install(|| {
        for batch in db_items_rx {
            batch.into_par_iter().try_for_each(|mut item| {
                let result = scanner.reprocess(&mut item)?;
                processed_tx.send(processed_item)
                Ok(())
            })?;
        }

        Ok(())
    })?;

    Ok(())
}

The ThreadPool::install function takes a closure, and registers the thread pool so that any uses of Rayon inside that closure will use it instead of the global thread pool.

With that small change, everything worked properly.

Alternative Solutions

For a data pipeline like this, the auto-parallelism of Rayon is arguably not even a good fit, since the pipeline stages are long-lived and well defined. Another solution would be to manually create the threads for each pipeline stage.

Since channel senders and receivers in flume (and most other channel libraries) can be cloned while referencing the same channel, this doesn’t take much effort, and would look something like this:

std::thread::scope(|scope| {
    const NUM_THREADS: usize = 4;
    let stage_tasks = itertools::repeat_n((db_items_rx, processed_tx), NUM_THREADS)
        .into_iter()
        .map((|db_items_rx, processed_tx)| {
            scope.spawn(|| reprocess(db_items_rx, processed_tx))   
        })
        .collect::<Vec<_>>();

    // Spawn other tasks, handle thread results, etc.
});

Takeaways

Thread pools are great, but can cause problems when multiple pieces of code with control flow dependencies are sharing the same pool.

Use the debugger as a first resort! VS Code’s Rust debugging support works great. I believe NeoVim’s support is quite good now too, but I haven’t tried it yet.

Switching Result and Option in Rust with Transpose

Daniel Imfeld — Thu, 05 Jan 2023 00:00:00 +0000

When writing Rust code, it’s common to have an Option value that you want to map through some fallible operation. For example, you have an Option<String> and you want to parse it as an integer. You could do something like this:

fn parse_maybe_int(s: Option<String>) -> Option<Result<i32, ParseIntError>> {
    s.map(|s| s.parse::<i32>())
}

But this gets awkward if you want to return an error when the parsing fails, but don’t want to unwrap the Option just yet. When I first started writing Rust I would use the match syntax to take care of it.

let input = Some("3".to_string());
let value = match parse_maybe_int(input) {
    Some(Ok(value)) => value,
    Some(Err(e)) => return Err(e),
    None => None,
};

One of Rust’s best features is the ? operator, which allows you to unwrap a Result and return if it’s an error. So having to write this match every time gets old quick.

Fortunately, there’s a method in the standard library that addresses this exact problem. When you have anOption<Result<T, _>> or a Result<Option<T>, _> you can use the transpose method to swap the Result and Option.

let input = Some("3".to_string());
let value = parse_maybe_int(input).transpose()?;

And then from there, you can build the transpose straight into the method:

fn parse_maybe_int(s: Option<String>) -> Result<Option<i32>, ParseIntError> {
    s.map(|s| s.parse::<i32>()).transpose()
}

let input = Some("3".to_string());
let value = parse_maybe_int(input)?;

Much better. I haven’t seen transpose mentioned much, but I’ve found it very useful, so I hope this helps someone.

Process Streaming Uploads with Axum

Daniel Imfeld — Sat, 26 Nov 2022 00:00:00 +0000

Many web services allow the user to upload a file and save it to some other place such as Amazon S3. Commonly, you can tell S3 to generate a special presigned URL which the client can use to upload the file to a specific location.

But sometimes you also want to be able to process the data as it streams through, which means it’s easiest to handle it yourself.

I had a need for this in my Pic Store project recently. When the user uploads an image, it passes the data through to the destination backing store, such as S3, but it also needs to calculate a hash of the image and figure out the format and dimensions.

This project uses the Rust Axum framework, but the same techniques work across other frameworks as well.

Receive the request as a stream

First, we set up our route handler with a streaming reader. Axum provides a BodyStream extractor which does the hard work for us.

pub async fn upload_image(
  // Other data -- app state, user info, etc.
  State(state): State<AppState>,
  Authenticated(user): Authenticated,
  Path(image_id): Path<BaseImageId>,
  // The actual request body
  stream: BodyStream,
) -> Result<impl IntoResponse, Error> {
  let image_location = get_base_image(&state, &user, image_id).await?;
  todo!();
}

pub fn configure_router() -> Router {
    Router::new()
        .post('/images/:image_id/upload', upload_image)
        // And so on...
}

The handler starts by reading the database to figure out where the image should be stored, if the user has permission to upload this image, and so on. I’ll skip over the details here since they’re not really related, which means it’s time to look at the data.

Setting up the Destination Writer

There’s a lot of ways to handle the S3 write, but my application needed something that could potentially talk to multiple storage services. The object_store crate is a good choice here. It’s part of the Apache Arrow project so it sees active development, and I found it very easy to use.

Since we’re streaming the data, we’ll use a multipart upload which removes the need to buffer the data locally. The object_store transparently manages the buffers and metadata around the multipart upload process.

let store = object_store::aws::AmazonS3Builder::new()
  .with_bucket_name(bucket)
  // Auth options go here too if you're using a fixed access key and secret.
  .build();

let (upload_id, mut writer) = operator.put_multipart(image_location).await?;
let upload_result = handle_body(&mut writer, stream).await;

todo!("Handle the result");

/// Handle the request body
async fn handle_body(
  writer: &mut Box<dyn AsyncWrite + Unpin + Send>,
  mut stream: BodyStream,
) -> Result<(String, ImageInfo), Error> {
  todo!();
}

The put_multipart function returns an object that implements AsyncWrite, and also an upload ID. The upload ID can be used to cancel the multipart upload in case of an error.

We’ll write a separate handle_body function to do the actual work here, so that we can easily catch any error that occurs and perform the cancellation.

Handling the Data

Now that the writer is ready, we can also set up our other operations – calculating the hash and figuring out the image format and dimensions.

Accumulating the header

I used the imageinfo crate here to examine the image, and it requires just the first 1024 bytes of the file to accomplish this.

There’s a good chance that the first chunk of the body will be larger than 1024 bytes, but we can’t count on that, so we use a Header object to accumulate data until we have enough.

This is actually the most complicated block of code in this article, but all it’s really doing is filling up a buffer and trying not to make more copies than it needs to. I’m including this since you might find it useful, but feel free to skip to the next section if you don’t care about the details here.

const HEADER_CAP: usize = 1024;

struct Header {
  buf: HeaderBuf,
}

enum HeaderBuf {
  Empty,
  Vec(Vec<u8>),
  Ref(Bytes),
}

impl Header {
  fn new() -> Header {
      Header {
          buf: HeaderBuf::Empty,
      }
  }

  fn as_slice(&self) -> &[u8] {
      match &self.buf {
          HeaderBuf::Vec(vec) => vec.as_slice(),
          HeaderBuf::Ref(bytes) => bytes.as_ref(),
          HeaderBuf::Empty => panic!("as_slice on empty header"),
      }
  }

  /// See if the buffer is full enough
  fn ready(&self) -> bool {
      let len = match &self.buf {
          HeaderBuf::Vec(vec) => vec.len(),
          HeaderBuf::Ref(bytes) => bytes.len(),
          HeaderBuf::Empty => 0,
      };

      len >= HEADER_CAP
  }

  /// Add more data to the header buffer.
  fn add_chunk(&mut self, bytes: &Bytes) {
      if self.ready() {
          return;
      }

      self.buf = match std::mem::replace(&mut self.buf, HeaderBuf::Empty) {
          // This is the first chunk.
          HeaderBuf::Empty => HeaderBuf::Ref(bytes.clone()),
          // It's already a Vec, so add an additional chunk to the Vec.
          HeaderBuf::Vec(mut vec) => {
              let needed = HEADER_CAP - vec.len();
              let actual = needed.min(bytes.len());
              vec.extend(bytes.slice(0..actual));
              HeaderBuf::Vec(vec)
          }
          HeaderBuf::Ref(first_bytes) => {
              // We had a first chunk of bytes but it wasn't big enough, so
              // create a new Vec from the first chunk...
              let mut vec = Vec::with_capacity(HEADER_CAP);
              vec.extend(first_bytes.iter().take(HEADER_CAP));

              // And then add the current chunk
              let needed = HEADER_CAP - vec.len();
              let actual = needed.min(bytes.len());
              vec.extend(bytes.slice(0..actual));

              HeaderBuf::Vec(vec)
          }
      };
  }

  /// Try to get the image information.
  fn parse(&self) -> Result<ImageInfo, ImageInfoError> {
      assert!(self.ready());
      let bytes = self.as_slice();
      ImageInfo::from_raw_data(bytes)
  }
}

Consuming the Stream

With our header accumulation ready, we can fill in our handle_body function and actually consume the stream. First, we’ll initialize our other data handling. I used blake3 for the hash and the aforementioned imageinfo for the image processing.

let mut hasher = blake3::Hasher::new();

let mut header = Header::new();
let mut info: Option<ImageInfo> = None;

The Stream abstraction provides a few ways to consume the data. For our purposes, a while loop is easiest. Each chunk then goes through the hasher and the image header object, and finally out to the writer.

while let Some(chunk) = stream.try_next().await? {
  hasher.update(&chunk);

  if info.is_none() {
    header.add_chunk(&chunk);
    if header.ready() {
      let i = header.parse()?;
      info = Some(i);
    }
  }

  writer.write_all(&chunk).await?;
}

Finishing up

Once the loop is done, we just have to clean things up. First we’ll look at the results of our data processing tasks and return from handle_body.

// Unwrap the `Option`, returning an error if we never saw enough data to fill the buffer.
let info = info
  .ok_or(Error::ImageHeaderDecode(ImageInfoError::UnrecognizedFormat))?;

// Get the calculated hash.
let hash = hasher.finalize();
let hash_hex = hash.to_string();

Ok((hash_hex, info))

Finally, we update our original call to handle_body to examine its result.

let upload_result = handle_body(&mut writer, stream).await;
let (hash_hex, info) = match upload_result {
  Ok(data) => {
    writer.shutdown().await?;
    data
  }
  Err(e) => {
    store
      .abort_multipart(&image_location, &upload_id)
      .await
      .ok();
    return Err(e);
  }
};

If everything succeeded, we call shutdown, which closes the stream and ensures that all the data has been written. object_store requires that you explicitly shutdown the stream, since it also uses this opportunity to tell S3 that the upload is complete.

If something failed, then we just tell the store to abort the upload, which lets S3 know that it should purge the data uploaded so far.

And then we’re done! My actual application code goes on to save the resulting metadata to the database and set up some background jobs to convert the image to other formats.

Handling streaming request bodies can feel like a daunting task. But I hope this has demonstrated how the Rust ecosystem provides great tools for making this an ergonomic and relatively painless process.

Setting up Let's Encrypt with NGINX in Docker

Daniel Imfeld — Thu, 17 Nov 2022 00:00:00 +0000

I was setting up Let’s Encrypt today for the first time, and it’s impressive how easy they’ve made this process that used to be a big hassle. Our NGINX configuration at work is too complicated for certbot’s built-in configuration management, and so we write the configuration ourselves.

NGINX Configuration

First, you need to respond to the ACME request on non-secure HTTP. This configuration does that while redirecting everything else to HTTPS.

server {
  listen 80;
  server_name _;
  # The directory that certbot will use to respond to the ACME challenge
  root /var/www;

  location /.well-known/acme-challenge {
    try_files $request_uri =404;
  }

  location / {
    rewrite ^ https://$host$request_uri? permanent;
  }
}

After that, you can set up your server as normal, but one problem remains…

SSL Certificate Bootstrapping

You don’t have the SSL certificates yet, and NGINX won’t start if they aren’t present.

This would present a chicken-and-egg problem, but we can supply a fake chicken in the form of a self-signed certificate.

Browsers will reject this, but that’s ok. All we really need to do is get NGINX to start so it can serve the port 80 ACME requests, then certbot will be able to function.

You could also just comment out the server block that needs the SSL certificate, but that is more of a hassle IMO.

Since I always forget how to generate a simple self-signed SSL cert, here’s one method:

openssl req -nodes -batch -x509 -newkey rsa:2048 \
  -keyout privkey.pem -out fullchain.pem -days 360

Then you can copy the resulting files into the proper location in /etc/letsencrypt/live/$CERTNAME/, and start NGINX.

Now, certbot doesn’t really like if there’s some fake self-signed certificate in the directory it’s expecting to use, so once NGINX is started you’ll want to delete the files you just created.

sudo rm -i /etc/letsencrypt/live/$CERTNAME/*.pem
sudo rmdir /etc/letsencrypt/live/$CERTNAME

Finally, you can run certbot certonly --nginx --cert-name $CERTNAME -d $DOMAIN to generate the certificate.

NGINX inside Docker

If you’re running NGINX inside a Docker container, you should use certbot’s webroot method instead of the nginx method.

certbot certonly \
  --webroot -w /var/www \
  --key-type ecdsa \
  --cert-name $CERTNAME \
  -d $DOMAIN

In this case you’ll also need to tell the NGINX in the docker container to reload, and certbot makes this easy with post-renewal hooks.

Create the file /etc/letsencrypt/renewal-hooks/post/nginx-docker-reload.sh with these contents:

#!/bin/bash
set -e
docker exec -it nginx nginx -s reload
echo Reloaded nginx!

You can test this script by running sudo certbot renew --dry-run.

If you need to copy the SSL certificates from /etc/letsencrypt/live/$CERTNAME to somewhere else that the Docker container can see, you can also do it in that hook.

Alternative Methods

I should note that ACME/LetsEncrypt/certbot can also work via DNS records. For this particular case that was not an option, but it could make things easier, and certbot has a number of plugins to interface with various DNS providers.

Starting with Solana, Part 4 - A Todo List with Rewards

Daniel Imfeld — Fri, 05 Nov 2021 00:00:00 +0000

In part 3 we looked at testing and deploying our Solana program. I was going to put together an article about using Svelte with Solana for part 4, but that will wait for part 5 instead while we focus on making a more interesting Solana program – a todo list where anyone can add an item and attach a reward for completion.

Program Structure

The program workflow will look something like this:

Someone creates a list.
A second person adds an item to the list.
When the task is done, the list owner marks it finished.
The person who added the item also marks it finished.
The token balance on the item is transferred to the list owner.

Our program will have four instructions to manage a list and its items:

NewList to make a new todo list.
Add to add an item to a list.
Cancel to delete an item, unfinished, and return the reward to the item creator.
Finish to remove an item and transfer the reward when both the list’s owner and the item’s creator agree it’s finished.

The Finish instruction’s escrow mechanism is intentionally simple to avoid overloading this post with new information.
Real programs would want to use something more secure. paulx's blog has a thorough explanation of how to do it right.

Program-Derived Addresses

With this program, each todo list is a Solana account, and each item in the list is also an account. Anyone can add an item with an attached token balance, and when the list owner and the item creator both mark it finished, the item’s balance transfers to the list owner’s account as a reward.

Normally, to use such a list you would need to know two public keys, one for the list’s owner and another for the list itself. But Solana lets us use program derived addresses (PDAs) as a better solution. These addresses are calculated from a hash of the program ID and various “seed” values which can include public keys or other strings of bytes, up to 32 bytes each.

Don’t worry if this seems complex right now. PDAs are a big part of Solana, but once we use them below it will make more
sense.

Solana also requires that program derived addresses can not have an associated private key, and it enforces this by checking that that they are not valid values on the ed25519 elliptical curve used to generate keys. After calculating the hash above, a “bump” value is added to the hash until a value is found that is not on the curve. This bump starts at 255 and is decremented until it finds a value not on the curve, and that becomes the actual address.

It’s not guaranteed that this “check and decrement” bump process will generate a valid PDA, but the chances are very high, and importantly it is deterministic – given a program ID and seeds, the same PDA will always be derived. And since the address is not a real public key, it’s guaranteed that the program from which the PDA came is the only thing that can generate a signed transaction from it.

Programs generate signed transactions for a PDA by attaching the seeds and bump value to an invocation that uses it. The Solana runtime can then use this information to verify that the PDA actually belongs to the program.

Returning Errors

The program may need to return some custom errors when things go wrong. Anchor has an #[error] macro to allow this, and we can place it on an enum to create the error values and associated error messages.

The error functionality here is not very expressive; the Solana program itself can only return a number, and so the error messages themselves are stored in the generated Anchor IDL file and translated by the Anchor runtime. This means that your error messages can only be static strings, without any associated data or other formatting capability.

#[error]
pub enum TodoListError {
    #[msg("This list is full")]
    ListFull,
    #[msg("Bounty must be enough to mark account rent-exempt")]
    BountyTooSmall,
    #[msg("Only the list owner or item creator may cancel an item")]
    CancelPermissions,
    #[msg("Only the list owner or item creator may finish an item")]
    FinishPermissions,
    #[msg("Item does not belong to this todo list")]
    ItemNotFound,
    #[msg("Specified list owner does not match the pubkey in the list")]
    WrongListOwner,
    #[msg("Specified item creator does not match the pubkey in the item")]
    WrongItemCreator,
}

Test Infrastructure

Of course, we also need tests for all this. I put together a few utility functions to perform common operations which you may also find useful. Feel free to copy them into your own tests if you’d like.

First, we define our imports and set up the Anchor provider and program.

const anchor = require('@project-serum/anchor');
const BN = require('bn.js');
const expect = require('chai').expect;
const { SystemProgram, LAMPORTS_PER_SOL } = anchor.web3;

const provider = anchor.Provider.env();
anchor.setProvider(provider);
const mainProgram = anchor.workspace.Todo;

Our tests need to check that the program is transferring balances around properly, so the getAccountBalance function handles that. User accounts also pay small transaction fees, so the expectBalance function wraps the proper call to check a balance with a bit of slack to account for that. Each of these functions is small, but together they are called about 50 times so the savings and convenience adds up.

async function getAccountBalance(pubkey) {
  let account = await provider.connection.getAccountInfo(pubkey);
  return account?.lamports ?? 0;
}

function expectBalance(actual, expected, message, slack=20000) {
  expect(actual, message).within(expected - slack, expected + slack)
}

Most of the tests simulate multiple users interacting with a contract. For this we have thecreateUser function which makes a user account with an airdropped balance and an anchor Providerfor the user. The createUsers function makes multiple users in parallel to speed up tests a bit.

Each Anchor Program instance is linked to a particular Provider, which is in turn linked to a particular user and wallet. To make it easier to call a program from various users, the programForUser function creates a new Program instance, based on mainProgram but with the given user’s Provider.

async function createUser(airdropBalance) {
  airdropBalance = airdropBalance ?? 10 * LAMPORTS_PER_SOL;
  let user = anchor.web3.Keypair.generate();
  let sig = await provider.connection.requestAirdrop(
    user.publicKey, airdropBalance);
  await provider.connection.confirmTransaction(sig);

  let wallet = new anchor.Wallet(user);
  let userProvider = new anchor.Provider(provider.connection,
    wallet, provider.opts);

  return {
    key: user,
    wallet,
    provider: userProvider,
  };
}

function createUsers(numUsers) {
  let promises = [];
  for(let i = 0; i < numUsers; i++) {
    promises.push(createUser());
  }

  return Promise.all(promises);
}

function programForUser(user) {
  return new anchor.Program(mainProgram.idl, mainProgram.programId,
    user.provider);
}

The full test file can be found on this Github repository and I’ll walk through the parts relevant to specific instructions in later sections.

Creating a List

Anyone can run the NewList instruction to create a new todo list. The TodoList account structure holds all the information for a list, including the list’s owner and name, how many items it can hold, and references to the accounts for each item in the list. Finally it also holds the bump value that was used to calculate its PDA, which feeds into Anchor’s autogenerated verification processes.

The address for the todo list is calculated from a combination of the owner’s pubkey and the name of the list, so a user could also create different lists with different names.

TodoList Data Structure 🔗

#[account]
pub struct TodoList {
    pub list_owner: Pubkey,
    pub bump: u8,
    pub capacity: u16,
    pub name: String,
    pub lines: Vec<Pubkey>,
}

We also add a function to calculate the space needed for a TodoList structure. We could do this all within Anchor’s account macro but this is much easier to manage. Remember that accounts that don’t have any dynamically-sized data (Strings or Vecs) can just #[derive(Default)] to have Anchor automatically calculate the size.

impl TodoList {
    fn space(name: &str, capacity: u16) -> usize {
        // discriminator + owner pubkey + bump + capacity
        8 + 32 + 1 + 2 +
            // name string
            4 + name.len() +
            // vec of item pubkeys
            4 + (capacity as usize) * std::mem::size_of::<Pubkey>()
    }
}

NewList Instruction Accounts

There’s a lot going on in the #[account] macro for the NewList instruction so we’ll go over them one by one.

fn name_seed(name: &str) -> &[u8] {
    let b = name.as_bytes();
    if b.len() > 32 { &b[0..32] } else { b }
}

#[derive(Accounts)]
#[instruction(name: String, capacity: u16, list_bump: u8)]
pub struct NewList<'info> {
    #[account(init, payer=user,
      space=TodoList::space(&name, capacity),
      seeds=[
        b"todolist",
        user.to_account_info().key.as_ref(),
        name_seed(&name)
      ],
      bump=list_bump)]
    pub list: Account<'info, TodoList>,
    pub user: Signer<'info>,
    pub system_program: Program<'info, System>,
}

The #[instruction] macro tells Anchor to deserialize the instruction arguments inside the autogenerated code so that the rest of the #[account] macros here can use their values. Note that Anchor doesn’t verify that the arguments listed in #[instruction] actually match the arguments in the implementation function.

init and payer are the same as before – they tell Anchor to create the specified account and who should pay for it.
space just calls TodoList::space, which we wrote above.
seeds tells Anchor that this account is a PDA and lists the seed values which are hashed to calculate it. Anchor generates code to rederive the PDA from the seeds and ensure that they match the passed-in value. Each seed value can be no more than 32 bytes, so the name_seed function helps with that.
bump is also used by Anchor’s PDA verification to avoid using a “check and decrement” loop to find the right bump value, which can be costly when operating under Solana’s strict computation budget. In most cases the caller will run the loop to find a proper bump value, since it tends to be operating in an environment such as a web browser where an extra for loop doesn’t matter much, and in many cases it would have already had the bump value on hand.

NewList Implementation

Finally, we write the actual implementation. Not much to do here; we just copy the instruction’s arguments into the TodoList data structure.

#[program]
pub mod todo {
    pub fn new_list(
        ctx: Context<NewList>,
        name: String,
        capacity: u16,
        account_bump: u8,
    ) -> ProgramResult {
        // Create a new account
        let list = &mut ctx.accounts.list;
        list.list_owner = *ctx.accounts.user.to_account_info().key;
        list.bump = account_bump;
        list.name = name;
        list.capacity = capacity;
        Ok(())
    }
}

Testing List Creation

Every test needs to create a list, and the createList function makes that reusable.

The list account is a PDA, so the first step is to calculate its value. We pass the program ID and the seed values to findProgramAddress, and it returns the calculated PDA and the bump value used to calculate it. From there, we can call newList. Finally, the function fetches the data from the initialized list and returns it.

async function createList(owner, name, capacity=16) {
  const [listAccount, bump] = await anchor.web3.PublicKey.findProgramAddress([
    "todolist",
    owner.key.publicKey.toBytes(),
    name.slice(0, 32)
  ], mainProgram.programId);

  let program = programForUser(owner);
  await program.rpc.newList(name, capacity, bump, {
    accounts: {
      list: listAccount,
      user: owner.key.publicKey,
      systemProgram: SystemProgram.programId,
    },
  });

  let list = await program.account.todoList.fetch(listAccount);
  return { publicKey: listAccount, data: list };
}

With that out of the way, the actual test is small. We create the user, create the list, and then check that all the fields of the empty list look like what we expect.

describe('new list', () => {
  it('creates a list', async () => {
    const owner = await createUser();
    let list = await createList(owner, 'A list');

    expect(list.data.listOwner.toString(), 'List owner is set')
      .equals(owner.key.publicKey.toString());
    expect(list.data.name, 'List name is set').equals('A list');
    expect(list.data.lines.length, 'List has no items').equals(0);
  });
});

Adding an Item

Anyone can call the Add instruction against any todo list. The arguments are the item name and the number of tokens for the item bounty. The account’s pubkey will also be added to the main todo list account.

ListItem Data Structure 🔗

Each list item keeps track of who created it and who has marked it finished. The reward amount is just the balance on the account, so we don’t need to explicitly track it here. As before, there’s a space function to better colocate the space calculation with the actual list of fields.

#[account]
pub struct ListItem {
    pub creator: Pubkey,
    pub creator_finished: bool,
    pub list_owner_finished: bool,
    pub name: String,
}

impl ListItem {
    fn space(name: &str) -> usize {
        // discriminator + creator pubkey + 2 bools + name string
        8 + 32 + 1 + 1 + 4 + name.len()
    }
}

Add Accounts

Once again, the list account includes the necessary seeds and bump to verify the PDA. We also pass the list owner’s pubkey and the list name to assist in verifying the PDA address. The item account initializes the new list with enough space to hold the name and other data.

Here we also see Anchor’s @ macro syntax. This can be used in a constraint to tell it to return a particular error instead of the default error for that constraint type.

Another new thing here is the way we define the list_owner field. Since we don't want to read the data
from the list_owner account, we can just use the AccountInfo type directly in the structure. Here
we just need it to assist in the PDA verification.

#[derive(Accounts)]
#[instruction(list_name: String, item_name: String, bounty: u64)]
pub struct Add<'info> {
    #[account(
      mut,
      has_one=list_owner @ TodoListError::WrongListOwner,
      seeds=[
        b"todolist",
        list_owner.to_account_info().key.as_ref(),
        name_seed(&list_name)
      ],
      bump=list.bump)]
    pub list: Account<'info, TodoList>,
    pub list_owner: AccountInfo<'info>,
    // 8 byte discriminator,
    #[account(init, payer=user, space=ListItem::space(&item_name))]
    pub item: Account<'info, ListItem>,
    pub user: Signer<'info>,
    pub system_program: Program<'info, System>,
}

Add Implementation

The add function both adds the new item to the list and attaches the reward amount to the item.

pub fn add(
    ctx: Context<Add>,
    _list_name: String,
    item_name: String,
    bounty: u64,
) -> ProgramResult {
    let user = &ctx.accounts.user;
    let list = &mut ctx.accounts.list;
    let item = &mut ctx.accounts.item;

    // Check that the list isn't already full.
    if list.lines.len() >= list.capacity as usize {
        return Err(TodoListError::ListFull.into());
    }

    list.lines.push(*item.to_account_info().key);
    item.name = item_name;
    item.creator = *user.to_account_info().key;

    // Move the bounty to the account. We account for the rent amount
    // that Anchor's init already transferred into the account.
    let account_lamports = **item.to_account_info().lamports.borrow();
    let transfer_amount = bounty
        .checked_sub(account_lamports)
        .ok_or(TodoListError::BountyTooSmall)?;

    if transfer_amount > 0 {
        invoke(
            &transfer(
                user.to_account_info().key,
                item.to_account_info().key,
                transfer_amount,
            ),
            &[
                user.to_account_info(),
                item.to_account_info(),
                ctx.accounts.system_program.to_account_info(),
            ],
        )?;
    }

    Ok(())
}

Anchor initializes an account with enough lamports to mark the account as rent-exempt, so we calculate how many more lamports should be transferred to equal the total reward amount, and invoke the transferinstruction on the system program to actually move the balance onto the item’s account.

Testing Item Addition

Once again, we’ll create a utility function to add items to a list. You could use a PDA here for the item account, but we don’t bother since the keys are stored in the list account anyway. This addItem function is similar to newList – we call the instruction to add the item to the list and then fetch the account data for both the list and the item.

async function addItem({list, user, name, bounty}) {
  const itemAccount = anchor.web3.Keypair.generate();
  let program = programForUser(user);
  await program.rpc.add(list.data.name, name, new BN(bounty), {
    accounts: {
      list: list.publicKey,
      listOwner: list.data.listOwner,
      item: itemAccount.publicKey,
      user: user.key.publicKey,
      systemProgram: SystemProgram.programId,
    },
    signers: [
      user.key,
      itemAccount,
    ]
  });

  let [listData, itemData] = await Promise.all([
    program.account.todoList.fetch(list.publicKey),
    program.account.listItem.fetch(itemAccount.publicKey),
  ]);

  return {
    list: {
      publicKey: list.publicKey,
      data: listData,
    },
    item: {
      publicKey: itemAccount.publicKey,
      data: itemData,
    }
  };
}

So first, a test to check that adding items actually works.

describe('add', () => {
  it('Anyone can add an item to a list', async () => {
    const [owner, adder] = await createUsers(2);

    const adderStartingBalance = await getAccountBalance(adder.key.publicKey);
    const list = await createList(owner, 'list');
    const result = await addItem({
      list,
      user: adder,
      name: 'Do something',
      bounty: 1 * LAMPORTS_PER_SOL
    });

    expect(result.list.data.lines,
      'Item is added').deep.equals([result.item.publicKey]);
    expect(result.item.data.creator.toString(),
      'Item marked with creator').equals(adder.key.publicKey.toString());
    expect(result.item.data.creatorFinished,
      'creator_finished is false').equals(false);
    expect(result.item.data.listOwnerFinished,
      'list_owner_finished is false').equals(false);
    expect(result.item.data.name,
      'Name is set').equals('Do something');
    expect(await getAccountBalance(result.item.publicKey),
      'List account balance').equals(1 * LAMPORTS_PER_SOL);

    let adderNewBalance = await getAccountBalance(adder.key.publicKey);
    expectBalance(adderStartingBalance - adderNewBalance, 
      LAMPORTS_PER_SOL,
      'Number of lamports removed from adder is equal to bounty');

    // Test that another add works
    const again = await addItem({ 
      list,
      user: adder,
      name: 'Another item',
      bounty: 1 * LAMPORTS_PER_SOL
    });
    expect(again.list.data.lines, 'Item is added')
      .deep.equals([result.item.publicKey, again.item.publicKey]);
  });
});

We should also test that any error conditions are properly checked. This test creates a list, fills it to capacity, and then checks that additional add operations fail.

it('fails if the list is full', async () => {
  const MAX_LIST_SIZE = 4;
  const owner = await createUser();
  const list = await createList(owner, 'list', MAX_LIST_SIZE);

  // Add 4 items, in parallel for speed.
  await Promise.all(new Array(MAX_LIST_SIZE).fill(0).map((_, i) => {
    return addItem({
      list,
      user: owner,
      name: `Item ${i}`,
      bounty: 1 * LAMPORTS_PER_SOL,
    });
  }));

  const adderStartingBalance = await getAccountBalance(owner.key.publicKey);

  // Now the list should be full.
  try {
    let addResult = await addItem({
      list,
      user: owner,
      name: 'Full item',
      bounty: 1 * LAMPORTS_PER_SOL,
    });

    console.dir(addResult, { depth: null });
    expect.fail('Adding to full list should have failed');
  } catch(e) {
    expect(e.toString()).contains('This list is full');
  }

  let adderNewBalance = await getAccountBalance(owner.key.publicKey);
  expect(adderStartingBalance,
    'Adder balance is unchanged').equals(adderNewBalance);
});

There are various ways to check an error condition, but the test here calls the failing function from within a try block, and then calls expect.fail to fail the test if addItem doesn’t throw an exception. In thecatch block we then check that the error message matches the one we expected to see.

In the actual fileI also tested creating an item with a too-small reward, which should also throw an error.

Cancelling an Item

Either the list’s owner or the creator of an item can cancel the item, in which case the reward tokens are returned to the item’s creator.

Cancel Instruction Accounts

This should look familiar by now. The main addition here is the item_creator account, which we need to transfer the item’s balance back to its creator. The address constraint in the account macro checks that the address of item.creator is the same as item_creator.

Also note that we need to mark item_creator with mut – although we don’t modify its data, transferring a balance to an account requires it to be writable. Solana validators use the “read-only vs. writable” distinction to allow multiple transactions to use accounts that are only used in read-only mode at a particular time. This is a similar concept to an RWMutex or Rust’s mutable borrow model, if you’re familiar with those.

From here on I’ll omit the seeds and bump from the account macro to save some space, but they are identical to the values in the Add instruction.

#[derive(Accounts)]
#[instruction(list_name: String)]
pub struct Cancel<'info> {
    #[account(mut, 
      has_one=list_owner @ TodoListError::WrongListOwner,
      ... omitted seeds and bump for brevity)]
    pub list: Account<'info, TodoList>,
    pub list_owner: AccountInfo<'info>,
    #[account(mut)]
    pub item: Account<'info, ListItem>,
    #[account(mut, address=item.creator @ TodoListError::WrongItemCreator)]
    pub item_creator: AccountInfo<'info>,
    pub user: Signer<'info>,
}

Cancel Implementation

The cancel instruction performs two checks:

The caller is either the list owner or the item creator
The cancelled item is actually in this todo list.

pub fn cancel(ctx: Context<Cancel>, _list_name: String) -> ProgramResult {
    let list = &mut ctx.accounts.list;
    let item = &mut ctx.accounts.item;
    let item_creator = &ctx.accounts.item_creator;

    let user = ctx.accounts.user.to_account_info().key;

    if &list.list_owner != user && &item.creator != user {
        return Err(TodoListError::CancelPermissions.into());
    }

    if !list.lines.contains(item.to_account_info().key) {
        return Err(TodoListError::ItemNotFound.into());
    }

    // Return the tokens to the item creator
    item.close(item_creator.to_account_info())?;

    let item_key = ctx.accounts.item.to_account_info().key;
    list.lines.retain(|key| key != item_key);

    Ok(())
}

Here we use item.close to tell Anchor/Solana to delete the account and also transfer any balance on it to the item_creator account. The final step is to actually remove the item from the list. Rust’s Vec::retain works similarly to a filter operation except it modifies the list in place.

Testing Cancellation

In the interest of space I'll only show the full code here for one of the passing tests. We create the list and item, do a quick check that it works right, and then cancel the item, checking again that the balance was returned and the item is removed.

it('List owner can cancel an item', async () => {
  const [owner, adder] = await createUsers(2);
  const list = await createList(owner, 'list');

  const adderStartingBalance = await getAccountBalance(adder.key.publicKey);

  const result = await addItem({
    list,
    user: adder,
    bounty: LAMPORTS_PER_SOL,
    name: 'An item',
  });

  const adderBalanceAfterAdd = await getAccountBalance(adder.key.publicKey);

  expect(result.list.data.lines,
    'Item is added to list').deep.equals([result.item.publicKey]);
  expect(adderBalanceAfterAdd,
    'Bounty is removed from adder').lt(adderStartingBalance);

  const cancelResult = await cancelItem({
    list,
    item: result.item,
    itemCreator: adder,
    user: owner,
  });

  const adderBalanceAfterCancel = await getAccountBalance(adder.key.publicKey);
  expectBalance(adderBalanceAfterCancel,
    adderBalanceAfterAdd + LAMPORTS_PER_SOL,
    'Cancel returns bounty to adder');
  expect(cancelResult.list.data.lines,
    'Cancel removes item from list').deep.equals([]);
});

In the actual test file we run the same test but with the list owner cancelling the item, and then there are three error checking tests:

Other users can not cancel an item – Similar to the above except we create a third user and try it have it cancel the item.
The item_creator account must match the creator field on the item.
The item must be in the list – Here we create two lists, add an item to one of them, but then pass the account of the other list when trying to cancel the item.

Finishing an Item

We’re almost done! The Finish instruction awards an item’s balance to the list owner when both the item creator and the list owner mark it finished.

Once again, a reminder that this is not a proper escrow implementation and there are all sorts of ways to exploit it.

Finish Instruction Accounts

Once again, we have the accounts for the list, the list owner, the item, and the user initiating the transaction.

#[derive(Accounts)]
#[instruction(list_name: String)]
pub struct Finish<'info> {
    #[account(mut, 
      has_one=list_owner @ TodoListError::WrongListOwner,
      ... omitted seeds and bump for brevity)]
    pub list: Account<'info, TodoList>,
    #[account(mut)]
    pub list_owner: AccountInfo<'info>,
    #[account(mut)]
    pub item: Account<'info, ListItem>,
    pub user: Signer<'info>,
}

Finish Implementation

The finish process sets the creator_finished flag or the list_owner_finished flag depending on who called it. If both of the flags are set, then the item is finished.

Finishing the item looks very similar to the cancel instruction, except we transfer the item’s reward to the list owner.

pub fn finish(ctx: Context<Finish>, _list_name: String) -> ProgramResult {
    let item = &mut ctx.accounts.item;
    let list = &mut ctx.accounts.list;
    let user = ctx.accounts.user.to_account_info().key;

    if !list.lines.contains(item.to_account_info().key) {
        return Err(TodoListError::ItemNotFound.into());
    }

    let is_item_creator = &item.creator == user;
    let is_list_owner = &list.list_owner == user;

    if !is_item_creator && !is_list_owner {
        return Err(TodoListError::FinishPermissions.into());
    }

    if is_item_creator {
        item.creator_finished = true;
    }

    if is_list_owner {
        item.list_owner_finished = true;
    }

    if item.creator_finished && item.list_owner_finished {
        let item_key = item.to_account_info().key;
        list.lines.retain(|key| key != item_key);

      item.close(ctx.accounts.list_owner.to_account_info())?;
    }

    Ok(())
}

Testing Finish

We can test the finish instruction using a very similar process. We create a list, add an item, and then try various ways to call finish.

The first test has the list owner call finish first, then the item creator.

it('List owner then item creator', async () => {
  const [owner, adder] = await createUsers(2);

  const list = await createList(owner, 'list');
  const ownerInitial = await getAccountBalance(owner.key.publicKey);

  const bounty = 5 * LAMPORTS_PER_SOL;
  const { item } = await addItem({
    list,
    user: adder,
    bounty,
    name: 'An item',
  });

  expect(await getAccountBalance(item.publicKey),
    'initialized account has bounty').equals(bounty);

  const firstResult = await finishItem({
    list,
    item,
    user: owner,
    listOwner: owner,
  });

  expect(firstResult.list.data.lines,
    'Item still in list after first finish').deep.equals([item.publicKey]);
  expect(firstResult.item.data.creatorFinished,
    'Creator finish is false after owner calls finish').equals(false);
  expect(firstResult.item.data.listOwnerFinished,
    'Owner finish flag gets set after owner calls finish').equals(true);
  expect(await getAccountBalance(firstResult.item.publicKey),
    'Bounty remains on item after one finish call').equals(bounty);

  const finishResult = await finishItem({
    list,
    item,
    user: adder,
    listOwner: owner,
    expectAccountClosed: true,
  });

  expect(finishResult.list.data.lines,
    'Item removed from list after both finish').deep.equals([]);
  expect(await getAccountBalance(finishResult.item.publicKey),
    'Bounty remains on item after one finish call').equals(0);
  expectBalance(await getAccountBalance(owner.key.publicKey),
    ownerInitial + bounty, 'Bounty transferred to owner');
});

In the Github repository, we also test the opposite order of calls to finish, which should end in the same result.

The error checks here are similar to those for the cancel instruction:

Disallowed users can not call finish
The list owner account must match the owner recorded on the list
Calling finish on an item not in the list does not work.
Calling finish on an already-finished item does not work.

Next Steps

And that’s it! A lot of code here but hopefully it gives you a good idea on how a slightly more complex contract would fit together. In part 5 we’ll finally build a Svelte-based web interface to the todo list.

Hot Module Reloading for Content in Sveltekit

Daniel Imfeld — Mon, 01 Nov 2021 00:00:00 +0000

I recently converted my website from Sapper to SvelteKit. SvelteKit uses the Vite build tool, and one of Vite’s great features is Hot Module Reloading, or HMR, which can reload changed parts of a site without reloading the entire browser page.

This is great for speeding up developer productivity, and so I wanted to see if I could support HMR when changing the Markdown and HTML files that make up the content of this site as well. Between Vite’s HMR API and some SvelteKit features, it was easier than I expected.

Sometimes you may be able to just use Vite’s import.meta.glob to accomplish this. If this doesn’t fit your needs, read on!

Vite HMR Plugin

The first step is to add a build plugin into the Vite configuration. With SvelteKit, this configuration goes inside your svelte.config.js.

Vite plugins have a similar API to Rollup plugins, but with some extra methods. The plugins provide one or more hooks into the build process, and then Vite calls those hooks at the appropriate time.

First, our plugin provides a configureServer hook which tells Vite to watch the content directories for changes. In Rollup this would be done in the buildStart hook and call this.watch to add the paths, but that doesn’t work for Vite dev mode. Instead, you call server.watcher.add(path). In the snippet below, you can see I add three directories to the watcher.

Adding these directories tells Vite to monitor them for changes and have them participate in the HMR process. When a change occurs, Vite will call the handleHotUpdate hook with one argument of type HmrContext.

interface HmrContext {
  file: string;
  timestamp: number;
  modules: Array<ModuleNode>;
  read: () => string | Promise<string>;
  server: ViteDevServer;
}

The full documentation explains it in more detail, but you can then return modules, filter down the list of modules to reload fewer modules, or send a custom event to the client side using server.ws.send. In our case, we look for a path matching one of the content directories, and send a custom client event. The full plugin is below.

/** @type {import('@sveltejs/kit').Config} */
export default {
  kit: {
    vite: () => {
      plugins: [
        {
          name: 'watch-content',
          configureServer(server) {
            server.watcher.add(path.join(dirname, 'posts'));
            server.watcher.add(path.join(dirname, 'notes'));
            server.watcher.add(path.join(dirname, 'roam-pages'));
          },
          handleHotUpdate(ctx) {
            let m = /(notes|posts|roam-pages)\/(.*)\.(md|html)$/.exec(ctx.file);
            if (m) {
              let contentType = m[1];
              let id = m[2];

              // This is just a conversion from the directory
              // names to the URLs used in the site.
              if (contentType === 'roam-pages') {
                contentType = 'notes';
              } else if (contentType === 'posts') {
                contentType = 'writing';
              }

              ctx.server.ws.send({
                type: 'custom',
                event: 'content-update',
                data: {
                  type: contentType,
                  id,
                },
              });

              // Return an empty module list since we
              // handled it manually.
              return [];
            }

            // Not an event we care about, so just do
            // the default behavior.
            return ctx.modules;
          },
        },
      ];
    },
  },
};

The Client Side

Once we can send events to the client, we need to handle them and actually perform the reload. Happily, SvelteKit makes this easy to do. Each article page uses a SvelteKit load function to fetch the content, and SvelteKit also lets you force these load functions to rerun using the invalidate function.

Vite’s HMR API is fairly complex, but for our case of just listening to a custom event, it’s straightforward. We check for the presence of import.meta.hot to see if Vite is running, and if so it’s just a matter of adding the event listener for the custom content-update event.

<script>
  import { invalidate } from '$app/navigation';

  if (import.meta.hot) {
    import.meta.hot.on('content-update', (data) => {
      if (data.type === 'notes') {
        invalidate('/notes/list.json');
        invalidate('/notes/tags.json');
        invalidate(`/notes/note/${data.id}.json`);
      } else if (data.type === 'writing') {
        invalidate('/writing/list.json');
        invalidate(`/writing/${data.id}.json`);
      }
    });
  }
</script>

Here I just placed a single event listener at the root __layout.svelte file of the site, which examines the event and invalidates all the relevant endpoints. Larger sites would probably want to break this out to have each component manage its own HMR event handling, but the concept is the same.

SvelteKit internally tracks the URLs fetched by each load function, and to invalidate then compares its argument to the tracked URLs, and reruns any load functions on active pages or layout components that match. And that’s it!

Starting with Solana, Part 3 - Testing a Solana Program

Daniel Imfeld — Wed, 27 Oct 2021 00:00:00 +0000

In part 2, we took a look at the account macros that Anchor uses to get data in and out of your Solana program. Now let’s go back to Nader Dabit’s tutorial and finish our first program.

To recap, the Rust source for the program looks like this.

use anchor_lang::prelude::*;

declare_id!("Fg6PaFpoGXkYsidMpWTK6W2BeZ7FEfcYkg476zPFsLnS");

#[program]
pub mod p0001 {
    use super::*;
    pub fn create(ctx: Context<Create>) -> ProgramResult {
        let base_account = &mut ctx.accounts.base_account;
        base_account.count = 0;
        Ok(())
    }

    pub fn increment(ctx: Context<Increment>) -> ProgramResult {
        let base_account = &mut ctx.accounts.base_account;
        base_account.count += 1;
        Ok(())
    }
}

#[derive(Accounts)]
pub struct Create<'info> {
    #[account(init, payer=user, space= 8 + 8)]
    pub base_account: Account<'info, BaseAccount>,
    #[account(mut)]
    pub user: Signer<'info>,
    pub system_program: Program<'info, System>,
}

#[derive(Accounts)]
pub struct Increment<'info> {
    #[account(mut)]
    pub base_account: Account<'info, BaseAccount>,
}

#[account]
pub struct BaseAccount {
    pub count: u64,
}

Setting The Program ID

Again, the anchor build command will compile the program and generate a keypair that can be used to deploy it. Since Anchor’s template always has the same program ID at the top, we need to take the ID from the generated keypair and use it in our program.

The build output tells us where to find it.

The program address will default to this keypair (override with --program-id):
  /home/projects/solana/p0001/target/deploy/p0001-keypair.json

The keypair JSON file is actually an array of numbers, designed to conveniently translate into a byte array. But the solana CLI command can extract the public key – which is also the address – for us.

$ solana address -k target/deploy/p0001-keypair.json
6UrdK99AdvoFdM8WrteDfK9XS2ENSLEHFkXDSmMPryk

Here, your value should be different from the one I have. The address needs to go in two places:

The declare_id! macro at the top of our program’s source code.
The Anchor.toml file under the key with the same name as your program.

# Anchor.toml first section
[programs.localnet]
p0001 = "6UrdK99AdvoFdM8WrteDfK9XS2ENSLEHFkXDSmMPryk"

The keypair file will persist across builds, but a new one with a different address will be generated if you delete it.

Writing a Test

Anchor generates a test file for us, but you may remember from part 1 that it doesn’t do much. Nader’s article creates a test that calls both of the instructions and verifies that they work, so let’s walk through that.

The shell of the test imports the Anchor framework JavaScript files and gets the program ready to run. The tests default to using the mocha test framework, so the it function defines a test and describe can be used to group tests together.

const assert = require('assert');
const anchor = require('@project-serum/anchor');
const { SystemProgram } = anchor.web3;

describe('p0001', () => {
  const provider = anchor.Provider.env();
  anchor.setProvider(provider);
  const program = anchor.workspace.p0001;
  // To be filled in by the create test and used by the increment test.
  let _baseAccount;

  // Tests here
});

We then have two tests, one for each instruction. More complex instructions might require multiple tests.

Testing the Create Instruction

The first test calls the create instruction, then gets the data for the created account and verifies that it exists and has a zero value for count.

it('Creates a counter', async () => {
  const baseAccount = anchor.web3.Keypair.generate();
  await program.rpc.create({
    accounts: {
      baseAccount: baseAccount.publicKey,
      user: provider.wallet.publicKey,
      systemProgram: SystemProgram.programId,
    },
    signers: [baseAccount],
  });

  // The account should have been created.
  const account = await program.account.baseAccount.fetch(
    baseAccount.publicKey
  );
  console.log('Initial count: ', account.count.toString());
  assert.ok(account.count.toString() == 0);
  _baseAccount = baseAccount;
});

Notice that although the system program has a fixed ID, we do still have to pass its address into the program.

Be sure that you are not running a local copy of solana-test-validator, and then use anchor test to run your test.

Mocha needs to be installed globally, using npm install -g mocha or your package manager of choice. If you get the error below, that may be your problem.

$ anchor test
... other build output
> Failed to run test: mocha -t 1000000 tests/: No such file or directory (os error 2)

One oddity here is that count in our Rust program is a number, but the test calls toString() on it. Because JavaScript numbers are 64-bit floating point values, it can only accurately represent up to 53 bits worth of integers, so Anchor deserializes any value of 64 bits or above as a bn.js BigNum object.

> console.dir(account.count);
BN { negative: 0, words: [0, 0, 0], length: 1, red: null }

> console.log(account.count.toString());
0

Testing the Increment Instructions

The next test is for the increment instruction, which reuses the baseAccount from the first test.

it('Increments a counter', async () => {
  const baseAccount = _baseAccount;
  await program.rpc.increment({
    accounts: {
      baseAccount: baseAccount.publicKey,
    },
  });

  const account = await program.account.baseAccount.fetch(
    baseAccount.publicKey
  );
  console.log('After increment: ', account.count.toString());
  assert.ok(account.count.toString() == 1);
});

Not much to say about this one.

Deploy to Localhost

With solana-test-validator running on your local machine, you can deploy the contract.

$ anchor deploy
Deploying workspace: http://localhost:8899
Upgrade authority: /home/projects/.config/solana/id.json
Deploying program "p0001"...
Program path: /home/projects/solana/p0001/target/deploy/p0001.so...
Program Id: 6UrdK99AdvoFdM8WrteDfK9XS2ENSLEHFkXDSmMPryk

Deploy success

Next time we’ll start on a frontend to interact with our application. The big difference from Nader’s tutorial is that I’ll try to figure out how to do it in Svelte instead of React.

Starting with Solana, Part 2 - Anchor's Account Macros

Daniel Imfeld — Tue, 26 Oct 2021 00:00:00 +0000

Last time, we set up our Solana and Anchor toolchains, and started to create a simple Solana program, following Nader Dabit’s tutorial. The end of the post started to look into what a Solana “Account” actually is, so we’ll continue there with the Anchor framework’s#[account] macro.

There are actually 2 #[account] macros in Anchor. One is used on Account structures to facilitate reading and writing Accounts from their raw data into Rust structures. The other is used in conjunction with the Accounts trait derive macro, which is placed on structures used for instruction inputs.

`#[account]` on an Account Structure

The macro invocation on the BaseAccount struct marks it as actually being used as an account, and generates some code to copy between the raw account data and the Rust structure. From Solana’s point of view, an account is just a raw buffer of bytes. Anchor has its own AccountSerialize and AccountDeserialize traits, similar to (but simpler and more specialized than) the Rust serde crate’s Serialize and Deserialize traits.

With the cargo expand command we can take a look at the code generated for theBaseAccount struct, which looks like this.

#[account]
pub struct BaseAccount {
    pub count: u64,
}

First, the code to write the Rust BaseAccount structure to a raw account buffer.

#[automatically_derived]
impl anchor_lang::AccountSerialize for BaseAccount {
    fn try_serialize<W: std::io::Write>(
        &self,
        writer: &mut W,
    ) -> std::result::Result<(), ProgramError> {
        writer
            .write_all(&[16, 90, 130, 242, 159, 10, 232, 133])
            .map_err(|_| anchor_lang::__private::ErrorCode::AccountDidNotSerialize)?;
        AnchorSerialize::serialize(self, writer)
            .map_err(|_| anchor_lang::__private::ErrorCode::AccountDidNotSerialize)?;
        Ok(())
    }
}

It first writes a discriminator, 8 bytes taken from a SHA256 hash of the name of the struct (“BaseAccount” in this case). This adds a check that the data you’re about to read is actually a BaseAccount. I imagine it doesn’t do anything from a security standpoint since it’s easy to copy, but it at least catches mistakes in sending the wrong type of Account to the program.

You can pass a namespace string value into the account macro, which will be prefixed to the struct’s name when generating the discriminator to reduce the chances of your struct having the same discriminator as a struct from someone else’s program. This is in the source code but doesn’t appear to be documented, so I’m not sure if it’s a good idea to use this namespace feature right now.

Next, some code to actually write the data in the call to AnchorSerialize::serialize. The generated BorshSerialize trait implementation handles this. Borsh is a binary data serialization framework with an extra focus on security.

impl borsh::ser::BorshSerialize for BaseAccount
where
    u64: borsh::ser::BorshSerialize,
{
    fn serialize<W: borsh::maybestd::io::Write>(
        &self,
        writer: &mut W,
    ) -> ::core::result::Result<(), borsh::maybestd::io::Error> {
        borsh::BorshSerialize::serialize(&self.count, writer)?;
        Ok(())
    }
}

Since the structure has just one field, you can see it’s just writing self.count to the passed in writer.

Deserializing the struct from the raw account buffer is similar. It first checks that the discriminator value matches, then calls AnchorDeserialize::deserialize to read in the actual data.

#[automatically_derived]
impl anchor_lang::AccountDeserialize for BaseAccount {
    fn try_deserialize(buf: &mut &[u8]) -> std::result::Result<Self, ProgramError> {
        if buf.len() < [16, 90, 130, 242, 159, 10, 232, 133].len() {
            return Err(anchor_lang::__private::ErrorCode::AccountDiscriminatorNotFound.into());
        }
        let given_disc = &buf[..8];
        if &[16, 90, 130, 242, 159, 10, 232, 133] != given_disc {
            return Err(anchor_lang::__private::ErrorCode::AccountDiscriminatorMismatch.into());
        }
        Self::try_deserialize_unchecked(buf)
    }
    fn try_deserialize_unchecked(buf: &mut &[u8]) -> std::result::Result<Self, ProgramError> {
        let mut data: &[u8] = &buf[8..];
        AnchorDeserialize::deserialize(&mut data)
            .map_err(|_| anchor_lang::__private::ErrorCode::AccountDidNotDeserialize.into())
    }
}

impl borsh::de::BorshDeserialize for BaseAccount
where
    u64: borsh::BorshDeserialize,
{
    fn deserialize(buf: &mut &[u8]) -> ::core::result::Result<Self, borsh::maybestd::io::Error> {
        Ok(Self {
            count: borsh::BorshDeserialize::deserialize(buf)?,
        })
    }
}

Deriving the `Accounts` trait

A Solana program’s input is just a raw buffer of bytes containing account public keys and various information for the program, and the official Solana SDK provides a function to extract some basic information from that buffer.

pub unsafe fn deserialize<'a>(
    input: *mut u8
) -> (&'a Pubkey, Vec<AccountInfo<'a>, Global>, &'a [u8])

Anchor automatically calls this function and then goes a step further. Each instruction in an Anchor program has a related structure which defines the format of all the information, and the Accounts trait determines how this structure can be created from the result of the deserialize function. The derivation macro for this trait does a lot of heavy lifting, so we’ll skip some of the details for now.

The structure for the create instruction in the tutorial looks like this.

#[derive(Accounts)]
pub struct Create<'info> {
    #[account(init, payer = user, space = 8 + 8)]
    pub base_account: Account<'info, BaseAccount>,
    pub user: Signer<'info>,
    pub system_program: Program <'info, System>,
}

Anchor’s Accounts trait requires one function: try_accounts.

pub trait Accounts<'info>: ToAccountMetas + ToAccountInfos<'info> + Sized {
    fn try_accounts(
        program_id: &Pubkey, 
        accounts: &mut &[AccountInfo<'info>], 
        ix_data: &[u8]
    ) -> Result<Self, ProgramError>;
}

try_accounts takes an array of expected AccountInfo structures, and expects the implementor (almost always automatically done through the macro) to take the expected number of accounts from the front of the list and then change the reference to point to the remaining slice of the list. This feels a lot more like C than Rust, but given the Solana VM’s strict execution limits it makes sense, and everything using Anchor is autogenerating it anyway so the bug risk is fairly low. Both structures such as Createand the Account structures such as Account and Signer implement this trait and try_accounts.

In Create, base_account is the structure we looked at above, which holds the actual counter value. The Accounts derive macro provides its own version of an account macro that can specify information on each field of the structure, and the invocation on base_account has three arguments.

init tells Anchor that calling this instruction should create an account corresponding to the BaseAccount structure. The account is initialized with the 8-byte discriminator and the rest of the buffer is zeroed out.
payer indicates which account provides the tokens (called lamports) to pay for the newly-created account’s rent. Here, it’s the user account listed next in the structure. Anchor will transfer two year’s worth of lamports from the payer into the account so that it can be rent-exempt.
space is the number of bytes required to store the account’s data. This is specified when the account is created and can never be changed. Again, Anchor needs 8 bytes and then the rest is the space taken up by the . It would be nice if there was a way to automatically calculate this, but the macro may not have easy access to the actual BaseAccount structure definition here to do so.

Next, the user account is a Signer type. In Solana, a Signeris an account accompanied by a digital signature indicating that the account’s owner has authorized the transaction. (As opposed to someone else passing that account’s address into the program.) Signer’s implementation of try_accounts verifies that the referenced account also signed the transaction.

Finally, there is a reference to the system program. Solana’s system program provides various servicessuch as creating accounts, transferring lamports between accounts, and more.

The account macro has many other useful attributes, and some common ones are:

#[account(mut)] tells Anchor that your instruction may alter the account, so it should serialize the account structure back to the input buffer, where Solana will update its data if allowed.
#[account(signer)] verifies that an account is also the signer. The Signer structure type we used above also does this, but it doesn’t actually deserialize the account contents from the buffer, so this can be used if you need to use a type other than Signer.
#[account(close=target)] tells the program to close the account when the instruction finishes, and transfer its lamports to the account specified by target.
#[acccount(has_one=target)] enforces that the field name inside the account matches the account given with the same name in the Accounts structure. So in the above example, we could add a user field to BaseAccount, and then#[account(has_one=user)] on the base_account field would add a check that base_account.user == user.
#[account(owner=target)] is similar to has_one, but it checks that the account’s owner is equal to the account given in the field.

There are some more but these seem to be the most useful ones.

That’s it for this part. Next up, we’ll go back to implementing and testing the tutorial program.

How Svelte Makes Two-Way Binding Safe

Daniel Imfeld — Sat, 23 Oct 2021 00:00:00 +0000

When React was released, it famously omitted two-way binding between components. This style of binding automatically detects when an attribute on an element or component changes, and reflects that change into a local variable in your component where you can use it.

This seems convenient at first, but traditional implementations are error-prone, as it becomes difficult to keep the bound value and any derived data in sync.

The Svelte framework reintroduces first-class support for two-way binding, to the confusion of those who have long heard – and rightfully so – that it is a bad idea. But Svelte has certain other features that make it not only more convenient to use, but safe as well.

One-Way Binding

First, let’s take a look at one-way binding, since it is how React and “plain” JavaScript work. With this paradigm, attributes are set on elements and components in the normal HTML way, and changes in those attributes are handled by added event handlers to the components.

In Svelte, one-way binding might look like this:

<script>
  let value = 'Hello world!';
  let numWords = 2;

  function handleUpdate(e) {
    value = e.target.value;
    numWords = value.split(' ').filter(Boolean).length;
  }
</script>

<input type="text" {value} on:input={handleUpdate} />
<p>
  {numWords} words
</p>

Every time you type in the text field, an event is fired and handleUpdate recalculates the numWords variable. A bit boilerplate-y, but pretty easy. As more complex code is added with more components, handlers, and derived state, it can get a bit unwieldy, but most people consider this inconvenience worth it compared to the problems of two-way binding.

Two-Way Binding

Two-way binding works similarly behind the scenes. The syntax is more terse, and the framework automatically adds the event handler.

<script>
  let value = 'Hello world!';

  $: numWords = value.split(' ').filter(Boolean).length;
</script>

<input type="text" bind:value />
<p>
  {numWords} words
</p>

With traditional two-way binding, you don’t really have a way to know when value has been updated, and so it becomes difficult to make sure that any state derived from it is actually up to date. AngularJS (aka Angular 1) has a “digest” and “watchers” that rerun all template expressions in a component when something might have changed, and yes it does become a performance problem.

But for anything outside a template, you may be out of luck, and these bugs may manifest in weird ways where sometimes your derived state ends up with the right value and other times it does not, depending on what else is going on and if one of your code paths that happens to update the state gets run.

In AngularJS you can sometimes use $scope.watch to handle this, but you have to explicitly remember to use it where appropriate, and it can be easy to miss if your state happens to update due to other events half the time.

How Svelte Makes it Safe

So with all these issues, why is it safe in Svelte? You’ll notice in the example above, numWords is in an expression starting with $:. This prefix tells the Svelte compiler to track the dependencies of that expression (here, just value), and automatically rerun the expression any time one of its dependencies changes.

So any time value is updated, whether from a bind: directive or from other code in the component, the Svelte compiler’s generated code automatically recalculates numWords. Even better, if numWords starts to depend on other variables, Svelte picks up on that. You don’t have to keep track of which variables your expression relies on since the compiler does it for you.

Having experienced two-way-binding hell in AngularJS, I was also initially wary of it in Svelte. But having using it for over a year now I’ve found Svelte’s promises here to be true. There are still situations where you may want to use one-way binding, but they are intentional design choices in your code rather than a way to avoid consistency bugs.

The only caveat here is that you do have to remember to use the $: operator. This is true, but it’s not as big of a deal as it might seem to Svelte outsiders. The operator is such a core part of the framework that it becomes second nature after short while, and I find myself doing it by default, only putting thought into if I might not want to use it for a particular expression.

The exact mechanisms in the Svelte compiler to track dependencies and rerun expressions aren’t particularly complex, but they are out of the scope of a quick blog post. Svelte contributor Tan Li Hau has covered the topic in great detail, so I recommend his excellent series on Svelte internalsif you are interested.

Starting with Solana, Part 1

Daniel Imfeld — Fri, 22 Oct 2021 00:00:00 +0000

My involvement in the cryptocurrency space has been minimal, aside from mining some Ethereum back in 2015 and following trends from afar. As “web3” has gained more mindshare in the developer community, it feels like a good time to dip my toe in, so here’s an exploration of writing applications for the Solana blockchain. The format will be more stream-of-consciousness than my usual articles here, just writing things down as I discover them.

This article series will start out following Nader Dabit’s introduction and we’ll see where it goes from there.

Setup

You don’t need a wallet right away, but it’s an important part of interacting with any blockchain, so it’s a good place to start. I chose https://phantom.app/ which is a popular wallet browser extension for Solana. I have some Solana on the FTX exchange already and Phantom makes it really easy to transfer from FTX into your own account, so that’s nice.

The wallet gives you a recovery passphrase so that’s nice. Apparently this is standard nowadays, but when I first started playing with crypto you just had to make sure you didn’t lose your private key files so this is a nice change.

Next I installed the Solana CLI tools and the Anchor tools as linked from the article above.

Anchor is a Rust framework that provides some abstractions over the Solana VM.

Looks like there are three Solana cluster options:

localhost which just runs on your computer
devnet which is an actual Solana network, but for testing applications and where can give your wallet as many tokens as you want (devnet tokens aren’t real or worth anything)
mainnet beta, which is the current “real” network where real transactions happen

Once you have the Solana CLI tools installed, you can run solana-test-validator to start a single-node localhost cluster on your computer.

To start out with the testing code you’ll need to create a “paper wallet” as well. This isn’t really suitable from a security standpoint for real use but is fine for just testing. You can run solana address to get instructions on how to create that wallet if you don’t have it yet.

Nader’s article has a quick overview of other solana CLI commands that come in handy at the start. There are about 75 commands overall but it looks like most of them aren’t used too often.

The Anchor Project Template

Ok, so on to creating an Anchor project. anchor init p0001 is what I used. I initially called it just 0001 which Anchor will do, but Rust doesn’t let crate names start with numbers, so don’t do that.

So this gives you a scaffolding of a Rust Solana contract, a test file in JavaScript, and some miscellaneous stuff.

The Program

Opening the Rust file programs/p0001/src/lib.rs gives us this:

use anchor_lang::prelude::*;

 declare_id!("Fg6PaFpoGXkYsidMpWTK6W2BeZ7FEfcYkg476zPFsLnS");

 #[program]
 pub mod p0001 {
     use super::*;
     pub fn initialize(ctx: Context<Initialize>) -> ProgramResult {
         Ok(())
     }
 }

 #[derive(Accounts)]
 pub struct Initialize {}

So the program itself is a module inside the file, and each function in the module becomes an entrypoint into the contract. Solana calls each of these functions an “instruction.” The #[program] line is a macro that processes the module into a Solana program. You can build the project with anchor build.

It looks like Solana programs compile to a variation of the eBPF format. This is interesting since that’s a bytecode originally designed for high-performance packet filtering inside the kernel. I’ve been hearing of it used for other stuff here and there so I guess this is one example.

The IDL File

The build command generates the compiled program, a keypair file that can be used as a wallet to deploy the contract on, and an IDL file that describes the instructions exposed by the contract.

~/projects/solana/p0001 ❯ cat target/idl/p0001.json
{
  "version": "0.0.0",
  "name": "p0001",
  "instructions": [
    {
      "name": "initialize",
      "accounts": [],
      "args": []
    }
  ]
}

So as you see in the file above, this IDL file describes the initialize instruction generated by Anchor, which has no arguments or other associated data.

Let’s also take a look at the default test, at tests/p0001.js.

Testing

const anchor = require('@project-serum/anchor');

describe('p0001', () => {
  // Configure the client to use the local cluster.
  anchor.setProvider(anchor.Provider.env());

  it('Is initialized!', async () => {
    // Add your test here.
    const program = anchor.workspace.P0001;
    const tx = await program.rpc.initialize();
    console.log('Your transaction signature', tx);
  });
});

The Provider is an abstraction that rolls up the Solana network connection, the wallet, and a commitment, which basically defines how much confirmation you want on a block before your program can see it.

The test just gets the default Provider from the wallet and cluster that you set up with the anchor command. Real programs would get the user’s wallet details and use that.

So then the test makes gets a reference the to program, calls the program’s initialize instruction, and returns the signature of the transaction for that call. Apparently Solana is able to wrap up multiple instruction calls into a single transaction, even to different programs, which is interesting.

Building an Actual Project

The next step in Nader’s article is to build a very simple program with just a single counter. It has two instructions, create and increment. Each one gets an Accounts structure that goes along with it.

#[program]
pub mod p0001 {
    use super::*;
    pub fn create(ctx: Context<Create>) -> ProgramResult {
        Ok(())
    }

    pub fn increment(ctx: Context<Increment>) -> ProgramResult {
        Ok(())
    }
}

#[derive(Accounts)]
pub struct Create<'info> {
    #[account(init, payer=user, space= 16 + 16)]
    pub base_account: Account<'info, BaseAccount>,
    #[account(mut)]
    pub user: Signer<'info>,
    pub system_program: Program<'info, System>,
}

#[derive(Accounts)]
pub struct Increment<'info> {
    #[account(mut)]
    pub base_account: Account<'info, BaseAccount>,
}

#[account]
pub struct BaseAccount {
    pub count: u64,
}

Accounts

Before we implement this, lets look at Accounts a bit more. The Solana Account documentation tells us that an account is not actually a wallet. Instead, it’s a way for the contract to persist data between calls. This includes information such as the count in BaseAccount, and also information about permissions on the account. It also has some lifetime information “expressed by a number of fractional native tokens called lamports.”

Accounts pay rent in the form of lamports, and if it runs out, then the account is purged from the blockchain. Accounts with two years worth of rent attached are “rent-exempt” and can stay on the chain forever. The current rent cost is designed to work out to 0.01 SOL per MB of data per day, or 3.65 SOL per MB per year. Since most accounts will hold far less than 1MB of data, the actual cost should usually be quite small.

That’s it for now. Next time, more on accounts, digging into the #[account] macro and the arguments to the create instruction, and actually implementing it.

Part 2 is up now!

Setting up SvelteKit with Storybook

Daniel Imfeld — Mon, 19 Jul 2021 00:00:00 +0000

I’ve become a big fan of using Storybook to develop components in an isolated context, and so it was a natural choice to use it for Ergo. In the process I found that some workarounds are needed to get things to work with SvelteKit, so this sums up everything I learned about getting it to actually work.

Once you have set up a SvelteKit project, the easiest way to add Storybook is with the npx sb init command. This installer will detect the project type and install all the necessary dependencies and configuration files.

Ok, let’s try it!

$ npm run storybook

> sveltekit-storybook@0.0.1 storybook sveltekit-storybook
> start-storybook -p 6006

info @storybook/svelte v6.3.4
info
ERR! Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: sveltekit-storybook/.storybook/main.js
ERR! require() of ES modules is not supported.

And more errors...

This is a common issue when using development tools with projects that set "type": "module" in the package.json file, as SvelteKit does. Storybook uses require to include .storybook/main.js, but that file is an ES Module due to the project settings, so it can not be loaded via require.

Fortunately, there’s a solution. Rename the file to .storybook/main.cjs, and Node.js will force it to be treated as a traditional “CommonJS” style module, that can be loaded with require. Storybook is set up to look for this extension, so it all works.

Trying again…

$ mv .storybook/main.js .storybook/main.cjs
$ npm run storybook
~/projects/sveltekit-storybook ❯ npm run storybook

> svelte-storybook@0.0.1 storybook sveltekit-storybook
> start-storybook -p 6006

info @storybook/svelte v6.3.4
info
ERR! Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: sveltekit-storybook/svelte.config.js
ERR! require() of ES modules is not supported.

Same issue, but slightly trickier. Storybook tries to be helpful and load the Svelte preprocessor configuration from your svelte.config.jsfile, but since we converted .storybook/main.js to be a CommonJS module, now it can’t require an ES module like svelte.config.js.

In this case we can’t just rename the file since SvelteKit complains loudly if the file is named svelte.config.cjs. The easiest solution I’ve found here is to just make the Storybook main.cjs file recreate the preprocessor configuration instead of pulling it in from svelte.config.js. Not ideal, but it’s a pretty small bit of configuration to duplicate, so not that big a deal.

const preprocess = require('svelte-preprocess');

module.exports = {
  // The rest of the config here...
  svelteOptions: {
    // Same options that you pass to preprocess in svelte.config.js
    preprocess: preprocess(),
  },
};

Ok, let’s try again.

$ pnpm storybook

> svelte-storybook@0.0.1 storybook sveltekit-storybook
> start-storybook -p 6006

info @storybook/svelte v6.3.4
info
info => Loading presets
WARN Unable to find main.js: sveltekit-storybook/.storybook/main
info => Loading 1 config file in "sveltekit-storybook/.storybook"
info => Loading 9 other files in "sveltekit-storybook/.storybook"
info => Adding stories defined in "sveltekit-storybook/.storybook/main.js"
WARN Unable to find main.js: sveltekit-storybook/.storybook/main
info => Using implicit CSS loaders
info => Using default Webpack4 setup

And more output...

Despite the “Unable to find main.js” warning, it works! Well, mostly. You can develop just fine for a while this way, but as soon as you use optional chaining or nullish coalescing, it falls apart. (That is, the .? or ?? operators.)

You can sort of get away with it by using Typescript to convert these features into older equivalents, but Svelte’s TypeScript support doesn’t currently process the template, so a component like this one will still have trouble.

<script lang="ts">
  export let value;
  export let defaultValue = 'N/A';
</script>

{value ?? defaultValue}

This is because Storybook uses Webpack 4 by default, which doesn’t support these newer JavaScript syntax features. A common solution here is to force Webpack to use a newer version of the acorn dependency, which it uses for parsing JavaScript. For Storybook, this causes very strange issues that mostly prevent Storybook from working at all.

A better solution is to use Webpack 5. Storybook recently gained full support for Webpack 5, so this can be enabled with just a few commands.

$ npm install --save-dev @storybook/builder-webpack5 @storybook/manager-webpack5

Once the dependencies are installed, a small update to our main.cjs will enable it.

module.exports = {
  core: {
    builder: 'webpack5',
  },
  svelteOptions: {
    preprocess: preprocess(),
  },
  // Rest of the configuration here
};

This doesn’t quite work though. I’ll skip all the errors but there are two things in the Webpack 5 configuration that need fixing:

All files must reference the same copy of the svelte library to avoid the dreaded “function called outside component initialization” error.
With "type": "module", the Webpack resolver must be set to fullySpecified: false, so that import calls don’t need to have the full file extension to work properly.

Storybook allows us to add a webpackFinal function to our configuration to make these changes.

const path = require('path');
const preprocess = require('svelte-preprocess');

module.exports = {
  core: {
    builder: 'webpack5',
  },
  svelteOptions: {
    preprocess: preprocess(),
  },
  webpackFinal: async (config) => {
    config.resolve = {
      ...config.resolve,
      alias: {
        ...config.resolve.alias,
        svelte: path.resolve(__dirname, '..', 'node_modules', 'svelte'),
      },
      mainFields: ['svelte', 'browser', 'module', 'main'],
    };

    config.module.rules.push({
      resolve: {
        fullySpecified: false,
        extensions: ['.js', '.ts'],
      },
    });

    return config;
  },
  // Rest of the config...
};

With all these changes, our Storybook compiles once again! But there’s one last error that shows up when loading Storybook in the browser. The devtools reveal the problem:

Uncaught ReferenceError: require is not defined
    at Object../.storybook/generated-stories-entry.js (generated-stories-entry.js:3)
    at __webpack_require__ (bootstrap:24)
    at __webpack_exec__ (main.iframe.bundle.js:221)
    at main.iframe.bundle.js:222
    at Function. __webpack_require__.O (chunk loaded:23)
    at main.iframe.bundle.js:223
    at webpackJsonpCallback (jsonp chunk loading:557)
    at main.iframe.bundle.js:1

Some code inside Storybook’s client also uses require and module, but when Webpack 5 sees a “module” type package it doesn’t provide all that CommonJS functionality. Fortunately, we can trick Webpack a bit here.

By adding the file .storybook/package.json with just the contents {} (yes, an empty object), Webpack 5 won’t see "type": "module" in the package.json at the root of your project, and will instead run in a module-agnostic mode. This allows files to use ES Module style import and export, but also provides require, module, and so on for CommonJS code that might need it.

Finally, we have a Storybook configuration where everything works. I’m sure that a lot of these workarounds will no longer be needed as time goes on, but for now I hope this helps you get a working setup.

I’ve created a GitHub repository with all these changes so that you can pull them into your own project.