DEV Community: Martin Hicks

Introducing Dynoxide: a fast, embeddable DynamoDB engine

Martin Hicks — Thu, 02 Apr 2026 21:02:51 +0000

I've been working with DynamoDB for the best part of a decade. It's my default database for most things I build at Si Novi, and I genuinely like it. The data modelling is satisfying once it clicks, the operational overhead is basically zero, and it scales without you having to think about it.

Last year, Si Novi started building Nubo, a native DynamoDB client for macOS and iPadOS, with Windows and Linux on the way. One of the features we wanted from the start was a built-in sandbox. A local DynamoDB instance running on-device that you could experiment with, no AWS credentials needed. Create tables, insert data, test access patterns. All offline. All on your machine. Maybe even on an iPad.

The problem was: how do you ship that?

DynamoDB Local's licence explicitly prohibits redistribution. You can't bundle it inside another application or sublicence it. Even setting the licence aside, the developer experience would have been awful. Asking users to separately download and install a Java-based emulator, with Docker or a JVM as a prerequisite, just to use one feature in your app? That's not a sandbox. That's homework.

And on iPadOS there's no path at all. No JVM. No Docker. Nothing.

So I started building something new. A DynamoDB-compatible engine in Rust, backed by SQLite, that compiles to a single native binary. Something I could embed directly into Nubo as a library.

That's how Dynoxide started.

From library to tool

The first version did exactly what I needed: a local DynamoDB that Nubo could bundle as its sandbox. Job done, in theory. But the thing that surprised me was just how small and fast it turned out to be.

A ~3 MB download (~6 MB on disk). Under 5 MB of RAM at idle. Cold startup in about 15 milliseconds, compared to over two seconds for DynamoDB Local. Numbers like that change what's practical. Instead of running one shared DynamoDB instance in your CI pipeline, you could spin up a fresh one per test. Isolated state. No cleanup step. No flaky tests from leftover data. Tear it down when you're done and the overhead is essentially nothing.

That realisation shifted what Dynoxide was becoming. It wasn't just infrastructure for Nubo any more. It was a tool in its own right.

The same properties that made it good for CI made it a natural fit for agentic development. A coding agent that needs to create tables, write data, or test queries can do all of that through Dynoxide's built-in MCP server. 34 tools, no setup required. Run dynoxide mcp and your agent has a full DynamoDB environment to work with.

And then I looked at how many Node.js projects still depend on dynalite for local DynamoDB. I owe a lot to dynalite. Using it through arc.codes and Architect is what turned DynamoDB from a database I'd reach for occasionally into the one I'd reach for first. Having a fast local emulator that just worked changed how I thought about building with DynamoDB. It made the feedback loop tight enough that I actually wanted to experiment. But dynalite hasn't seen updates in a while and newer DynamoDB features like transactions and streams aren't covered. Dynoxide supports both, along with the rest of the API surface. With an npm package shipping platform-specific binaries (the same approach esbuild and Biome use), it works as a drop-in replacement. Same workflow, one line change in your package.json.

Under the hood

Dynoxide implements the DynamoDB API as a translation layer in Rust, storing data in SQLite and compiling to a native binary with no runtime dependencies. DynamoDB Local takes the same general approach (SQLite as the storage engine is well documented) but ships as a Java application with a JVM dependency and a restrictive licence.

Because SQLite is an embedded database, Dynoxide doesn't have to run as a standalone server. It can be linked directly into another application, which is exactly how Nubo uses it. Encrypt the database file with SQLCipher and you've got a portable, encrypted, DynamoDB-compatible data store. Nubo uses this for its local cache: data you've previously accessed from AWS is kept in an encrypted database on-device, so it loads instantly and works offline.

To make sure Dynoxide actually behaves like the real thing, I built a conformance test suite. 526 tests, all validated against real DynamoDB on AWS. Every emulator gets the same suite. Dynoxide passes all 526. DynamoDB Local manages 92%. LocalStack gets 93%. Dynalite hits 81%.

The suite is public. Anyone can run it and verify.

Get started

Homebrew:

brew install nubo-db/tap/dynoxide

npm (drop-in dynalite replacement):

npm install --save-dev dynoxide

Cargo:

cargo install dynoxide-rs

Then just run it:

dynoxide

Full documentation is at dynoxide.dev, and the source is on GitHub.

Dynoxide is the foundation. But the thing I'm building on top of it is what I really want to talk about next. Nubo is a native DynamoDB client for macOS and iPadOS, and I think people who work with DynamoDB day to day will love it. If that sounds interesting, there's a mailing list at nubo.sinovi.uk.

Dynoxide stands on its own though. If you're running DynamoDB Local in CI and wondering why your pipeline is slow, or you're still using dynalite and need coverage for newer API features like transactions and streams, give it a try. I'd love to know what you think.

Si Novi will be at AWS Summit London on April 22nd. If you're there and want to talk DynamoDB, come and find me.

Using Whisper and FFmpeg to build a fully automated YouTube Shorts pipeline

Martin Hicks — Sat, 21 Mar 2026 22:13:53 +0000

My son Ben is 12. Back in August he decided he wanted to start a YouTube channel. We talked about what it could be and he landed on Premier League score predictions. Every gameweek he'd predict the scores for all 10 matches, record himself talking through his picks, then after the weekend we'd see how he did and record a review.

Simple concept. He was into it. I was into it. Let's go.

The manual era

The first video was recorded in the car on the drive home from France. I'd knocked together a Keynote presentation on my iPad with the fixtures for Gameweek 1. Team names, badges, blank spaces for Ben to write his predictions with Apple Pencil while screen recording. It was rough, but it worked. He recorded his picks, we uploaded it, and Ben's PL Predictions was born.

The workflow went like this: I'd create the Keynote slides manually, Ben would screen record on the iPad, I'd trim the video in iMovie, adjust the audio levels and brightness, then upload to YouTube Studio and fill in the title, description, and tags by hand.

For the review video I'd do the same thing again, but this time with actual results filled in and a little thumbnail showing what he'd predicted, so viewers could see both side by side as he talked through each result.

It was taking me about an hour on a Friday evening or Saturday morning. Not terrible, but I could feel the fragility of it. Ben was being consistent, recording every week without fail, and I didn't want me to be the reason he couldn't get a video out. If I was busy, or forgot, or just didn't fancy spending an hour creating slides and editing video, his streak would break. That didn't sit right.

JSON and Python: the first automation

After a couple of weeks I rebuilt the system. Instead of manually creating Keynote slides, I set up a JSON data model for each gameweek (fixtures, predictions, results, points) and wrote a Python CLI using Click that generates PowerPoint presentations from the data. The slides get copied to iCloud so they appear on the iPad automatically.

{
  "league": "EPL",
  "season": "2025-26",
  "gameweek": 30,
  "matches": [
    {
      "home": "ars", "away": "eve",
      "prediction": {"home": 2, "away": 0},
      "result": {"home": 2, "away": 0},
      "points": 3
    }
  ]
}

The scoring is simple: three points for an exact score, one point for the correct result (home win, away win, or draw), zero for wrong. The CLI generates prediction slides with blank scores for Ben to write on, and review slides with actual results, points, and a thumbnail of his predictions.

This got my weekly time down from an hour to about thirty minutes. Create the JSON with fixtures, run the command, copy to iCloud. After matches, add results, run the review command. Better, but I was still manually typing fixtures, manually watching his video to work out what he'd predicted, manually looking up results, manually editing in iMovie, and manually uploading to YouTube. Five of those steps are things a computer should be doing.

Automating everything else

I've just finished building the next phase, removing every remaining manual step. It's ready to go from the Gameweek 31 review and Gameweek 32 onwards. Here's what the pipeline looks like.

Fixtures from the FPL API

No more manual data entry. The Fantasy Premier League website exposes an undocumented API that the community has been using for years. It provides fixture data per gameweek, free, no authentication required.

$ python cli.py fetch-fixtures -w 31
Fetched 8 fixtures for GW31

The CLI maps FPL's numeric team IDs to our registry, handles postponed matches and writes the gameweek JSON automatically. It's worth noting this is an unofficial API with no formal terms of use for developers. For a personal project making a handful of calls per week, the risk is negligible, but it's not something I'd build a commercial product on.

Video transcription with Whisper

This is the one that surprised me most. After Ben records his prediction video, OpenAI's Whisper runs locally to transcribe the audio, then a regex parser extracts his predicted scores by matching them to fixtures by team name.

$ python cli.py transcribe -w 31
 1 | Bournemouth  | Man United  | 1 - 1  | matched
 2 | Brighton     | Liverpool   | 2 - 1  | matched
 ...
Matched 8/8 fixtures
Save these predictions to the gameweek JSON? [y/N]: y

It handles colloquial names like "Spurs," "Palace," and "Forest", and works out which score belongs to home and away based on context. I run the Whisper medium model with an initial prompt containing all 20 Premier League team names, which dramatically improves recognition accuracy. I tested it against Ben's existing recordings and it matched all predictions correctly: 10/10 for GW30 and 8/8 for GW31.

It asks for confirmation before writing to the JSON, so there's always a human check.

Results from the API

Same FPL endpoint, different data. After the weekend's matches:

$ python cli.py fetch-results -w 31
Updated 8 matches with results
Total points: 11

Merges results into the existing JSON, preserving Ben's predictions, and calculates points automatically.

Encoding publish-ready Shorts

This replaces iMovie entirely. A single command takes the raw iPad screen recording and produces a YouTube-optimised Short:

$ python cli.py encode-short -w 31 -t predict -i RPReplay_Final.MP4

Under the hood it runs a single Whisper pass (reused for both trim detection and SRT caption generation), detects the speech start via FFmpeg's silencedetect filter, finds the closing phrase ("see you next time") in the transcript, then encodes with normalised audio, proportional scaling and YouTube-recommended H.264 settings.

The trim is tight, half a second after the closing phrase, so it cuts before Ben swipes up to stop the screen recording and you don't see the iOS Control Centre.

Upload to YouTube

The final piece. One command uploads the encoded Short as a private video with auto-generated metadata:

$ python cli.py upload-short -w 31 -t predict
Title: GW31 Premier League Predictions #PremierLeague #matchweek31
Tags: Premier League, EPL, GW31, Football...
Thumbnail set
Captions uploaded
Privacy: PRIVATE

It generates the title, description (including season accuracy stats and an engagement question), tags with relevant team names, extracts and sets a thumbnail from the title frame, and uploads the SRT captions for search indexing.

Everything uploads as private. Ben and I review it in YouTube Studio and he publishes when he's happy. This was a deliberate decision. It's a child's channel and I wanted a human gate before anything goes public.

The YouTube API OAuth credentials are pulled from 1Password at runtime via the op CLI, so nothing sensitive sits on disk.

The weekly routine from Gameweek 32

What used to be nine manual steps should now be a handful of CLI commands plus Ben's two recordings:

fetch-fixtures then pptx-predict-table --icloud - pull fixtures, generate slides, send to iPad
Ben records prediction video
transcribe then encode-short then upload-short - process and publish
fetch-results then pptx-review-table --icloud - pull results, generate review slides
Ben records review video
encode-short then upload-short - process and publish
Review and publish manually in YouTube Studio

I'm looking forward to running it for the review once GW31's fixtures are all played, and both predictions and review for real in Gameweek 32 after the international break. If the numbers hold, my active time should go from an hour to under two minutes. And critically, none of those two minutes are things that block Ben. If I'm not around, someone else could run four commands, or I could do it from my phone over SSH. The next step is getting this set up on Ben's Chromebook so he can run the pipeline himself — once I work out why the option to enable Linux is disabled on his machine, he won't need me involved at all.

What I'm proud of

It's not the code. It's that Ben has stuck with it. He's at over a hundred subscribers, he's recorded every single gameweek this season, and he genuinely looks forward to it. He's learning that consistency matters more than virality, that showing up every week builds something, and that over a hundred people choosing to follow a 12-year-old's football predictions is pretty cool.

The automation exists to protect that consistency. Not to make the videos for him. He still records every one himself, writes his predictions by hand with Apple Pencil, and talks through his reasoning. The personality is all him. I just make sure the boring infrastructure never gets in the way.

If you're into Premier League predictions, or you want to support a kid who's been showing up every week, you can find Ben at youtube.com/@BensPLPredictions.

Recently Played: bringing back my Last.fm component

Martin Hicks — Sun, 08 Mar 2026 23:21:01 +0000

Around 2010, my personal website had a Last.fm widget showing what I'd been listening to. It was a small thing - just a few album covers and track names in the sidebar - but it was very me. Back then your personal site was an extension of yourself, and having your music taste ticking away in the corner felt right.

Fast forward fifteen years and I've rebuilt this site from scratch several times over. The Last.fm widget never made the cut again. Not for any good reason - it just fell off the list each time. When I was recently working on the sidebar I remembered it, and thought why not bring it back?

There's something I like about the full circle. The web has gone through its various phases since 2010 - the single page app years, the JavaScript-for-everything years - and come out the other side valuing progressive enhancement and server-rendered HTML again. My site is built with Eleventy and deployed from GitHub Actions to S3 with CloudFront. It felt fitting to bring back a feature from an earlier era, built with the principles I care about now.

So here's how it works.

The architecture

The component is a two-layer system: build-time SSR for the initial HTML, and client-side polling for live updates. It's dropped into the sidebar as a <now-listening> WebC element alongside <my-details>.

Build time:  listening.js → Last.fm API → tracks data → SSR into HTML
Runtime:     Client JS → polls Last.fm API every 60s → diffs → updates DOM
             ├── localStorage cache (2min TTL) for instant loads
             └── pauses when tab hidden, resumes on focus

Data layer: `src/_data/listening.js`

This is an Eleventy global data file that runs at build time. It calls the Last.fm API - specifically user.getrecenttracks - requesting the five most recent tracks for my account.

It has a three-second timeout so that if Last.fm is having a bad day, builds aren't left hanging. On any failure it returns { tracks: [] }, which means builds never break regardless of what the API does.

The raw API response gets mapped into clean objects - name, artist, album, url, art, and a nowPlaying boolean. All text fields are HTML-escaped and URLs are validated (only http: and https: schemes allowed) - this is the server-side XSS protection layer.

The data is then available to templates as listening.tracks via Eleventy's data cascade.

The component: `now-listening.webc`

The WebC component has three distinct parts.

Server-rendered HTML

A webc:type="render" script runs at build time, reading this.$data.listening.tracks. It generates the initial HTML so the page ships with real track data baked in. This means content is visible immediately on load - and for search engines or anyone browsing without JavaScript, this is the component. It's done. No spinner, no empty state.

A `<template>` element

An inert HTML template used by the client-side code for DOM cloning. It defines the track row structure with data-* attribute hooks - data-track, data-name, data-detail, data-art-placeholder, and data-now-playing. Nothing renders from this until JavaScript picks it up.

Client-side polling script

A self-executing IIFE (kept alive with webc:keep to prevent Eleventy from stripping it) that handles the live behaviour:

Polling - hits the Last.fm API every 60 seconds to keep the track list current.

localStorage cache - on page load, if a fresh cache exists (two-minute TTL), it renders from cache immediately and defers the first API call. This avoids the flash-of-stale-content problem on repeat visits or navigating between pages.

Diffing - compares JSON.stringify(tracks) against the last known state. If nothing has changed, the DOM stays untouched. No unnecessary reflows.

Visibility awareness - listens to visibilitychange to stop polling when the tab is hidden and restart when it becomes visible. If you leave the tab in the background for an hour, it's not hammering the API the whole time. Be a good citizen.

AbortController - cancels in-flight fetch requests when polling stops, so there's no risk of stale responses landing after the component has moved on.

Client-side XSS protection - uses the same safeUrl and esc helpers as the server layer. Content is inserted via textContent (inherently safe) and URLs are validated before being set as href or src attributes.

Design decisions

The main tradeoff worth calling out is the duplicated logic. The same API call and data mapping exists in both listening.js (server) and the client-side script. I could abstract it into a shared module, but the server code runs in Node during the Eleventy build and the client code runs in the browser. Keeping them self-contained means each layer is independently understandable and testable. The duplication is small and the mapping is straightforward - it's not the kind of logic that's likely to drift in dangerous ways.

The other thing I'm quietly pleased with is how little the component asks of the outside world. Between the visibility-aware polling, the localStorage cache, and the diff check before touching the DOM, it makes the minimum number of requests necessary. If you're pulling from someone else's API on every page load of your site, I think you owe it to them (and your users) to be thoughtful about it.

Was it worth it?

It took a couple of hours to build and it makes me smile every time I see it on the page. Sometimes the best features aren't the most technically ambitious - they're the ones that make a site feel like yours.

I might extract this into a standalone package at some point - the component is fairly self-contained and would slot into any Eleventy site with minimal config.

If you want to see it in action, it's in the sidebar at martinhicks.dev.

Remove trailing slash on Eleventy S3 hosted sites using Cloudfront function

Martin Hicks — Tue, 16 May 2023 00:00:00 +0000

Intro

This article will walk through the steps required to create a Cloudfront function to handle redirecting trailing slash URIs to non-trailing slash equivalents on your S3 hosted 11ty website.

Background

Several years ago we published our website at Si Novi using a hand-balled static site generator we built for ourselves, and deployed it to S3 with Cloudfront used for caching and routing our A record from Route 53.

For this old website we wanted to strip trailing slashes on URLs, so https://example.com/articles/some-article instead of https://example.com/articles/some-article/, personal preference I guess.

Anyway, to achieve this we used a Lamda@Edge function to handle the redirects for us - 301 redirecting from the trailing slash URI to the non-trailing slash URI - something we'd long achieved using .htaccess on an Apache server.

We published this function to the AWS Serverless Application Repository.

Fast forward a few years, and we now publish our website using Eleventy - still hosted on S3, still fronted with the Cloudfront CDN, but our long-standing redirect function no longer worked.

With 11ty we were hitting a redirect loop which we believe was due to it's use of subfolder index.html pages - our old hand-balled system created S3 objects like articles/some-article.html whereas Eleventy creates S3 objects like articles/some-article/index.html. The the old system resolved to the object correctly, whereas when using sub-directory within an index.html as 11ty and others do, this caused an infinite redirect loop.

Solution

1. Create a new CloudFront function

function handler(event) {
    var request = event.request;
    var uri = request.uri;

    var params = '';
    if(('querystring' in request) && (request.querystring.length > 0)) {
        params = '?'+request.querystring;
    }

    if(uri.endsWith('/')) {
        if(uri !== '/') {
            var response = {
                statusCode: 301,
                statusDescription: 'Permanently moved',
                headers:
                { "location": { "value": `${uri.slice(0, -1) + params}` } } // remove trailing slash
            }

            return response;    
        }


    }
    //Check whether the URI is missing a file extension.
    else if (!uri.includes('.')) {
        request.uri += '/index.html';
    }



    return request;
}

The above code achieves the same trailing slash removal as we had in our old Lambda@Edge function, but also includes an additional check to ensure that index.html is appended to any requests on their way to S3 (only if the request doesn't already include '.', so image/some-image.png will pass-through just fine ).

2. Publish the function

Save and publish your newly created function, in this example I've named it subfolder-index

3. Configure your Cloudfront distribution to route requests through the function

Modify your Cloudfront distribution's behaviour and set the published function to run on Viewer request.

Wrap up

That's it, you should now be seeing URLs redirected to your non-trailing slash URI preference, while still successfully serving the subfolder index.html file (which wont appear in the URL)

One benefit of using a Cloudfront function is it's cheaper to invoke than a Lambda@Edge function.

You can see our old Lambda@Edge function here, which still might be useful.

ps: If you're not hosting subfolder index.html files you can remove the else if from the Cloudfront function.

Enhance CSRF package - now supports multipart form data

Martin Hicks — Mon, 15 May 2023 00:00:00 +0000

I've just published a minor update to my CSRF plugin for Enhance projects.

v0.9.0 now supports multipart/form data, meaning you can easily use the <csrf-form></csrf-form> component for forms that contain file uploads.

usage:

<csrf-form method="post" action="/upload" enctype="multipart/form-data">
  <input type="file" name="file" />
</csrf-form>

outputs:

<form action="/si-novi/christopher-dee/media/upload" method="post" enctype="multipart/form-data">
  <input type="hidden" name="csrf" value="540c460e-946e-4c78-8c8d-63c4cd091ee9"> <!-- auto-generated hidden input with the unique csrf token for this request (use with verifyCsrfToken on your post handler) -->
  <input type="file" name="file">
</form>

Additionally, I've also improved the html generation for the component so you can now include all the following optional attributes and they'll pass-through into your HTML <form> element.

enctype
target
acceptCharset
autocomplete
id
novalidate
rel

Of course you can still use the standaone <csrf-input></csrf-input> if you'd prefer to use a standard form tag.

Get in touch

Any feedback on your usage of this plugin, bugs, or suggestions for improvements please get in touch on GitHub. I'd love to hear from you.

DynamoDB Streams locally with arc.codes & Enhance

Martin Hicks — Mon, 24 Apr 2023 00:00:00 +0000

Introduction

When working on a recent web app project using Enhance, I encountered a requirement for using both transactions and DynamoDB streams, two DynamoDB features that aren't supported by Dynalite - the fast in-memory DynamoDB engine that Architect uses to support local development.

In this article, I'll introduce my new plugin (arc-plugin-sandbox-stream) for working with DynamoDB streams locally within Arc and Enhance projects. This plugin enables developers to use DynamoDB streams locally in their arc or enhance sandbox environment.

Background

Architect got me hooked back in 2018 with its amazing local developer experience, it's a huge DX boon to be able to try ideas out on your own machine without being concerned about provisioning real infrastructure, or being tied to a location where network connectivity is guaranteed.

One of the services Architect spins up locally is Dynalite. It provides a fast in-memory DynamoDB engine. For lots of projects this engine is absolutely ideal, fast to launch, lightweight and not a process hog. But for this particular project I really needed streams and I wasn't prepared to sacrifice the local developer experience. So I set about researching how this could be achieved, if at all. Helpfully it's a question that has cropped up on Arc's discord and the thread of replies helped point me in the right direction.

I'd need to do a few things:

Switch to using DynamoDB Local
Create a middleware to poll the stream for new data
Invoke the corresponding function handler when new data is returned from the stream.

Using the plugin

The plugin kicks in when a @tables-streams pragma is discovered within your arc config, and queries the table meta data / stream meta data to retrieve iterators for each shard of the table's stream.

If data is found the plugin invokes the corresponding lambda function using arc's inbuilt invoke function.

Your function code will receive one or more results off the stream like so (as you'd expect if you've used streams on AWS):

[
  {
    "eventID": "7c1ac231-2c9d-4ba0-b3a2-1de2eb6602c0",
    "eventName": "MODIFY",
    "eventVersion": "1.1",
    "eventSource": "aws:dynamodb",
    "awsRegion": "ddblocal",
    "dynamodb": {
      "ApproximateCreationDateTime": "2023-04-24T12:19:00.000Z",
      "Keys": {
        "sk": {
          "S": "account:si-novi"
        },
        "pk": {
          "S": "account:si-novi"
        }
      },
      "NewImage": {
        "_type": {
          "S": "Organisation"
        },
        "name": {
          "S": "Si Novi"
        },
        "sk": {
          "S": "account:si-novi"
        },
        "created_at": {
          "S": "2023-04-24T12:19:07.409Z"
        },
        "id": {
          "S": "01GYSK850HWHGE36A2VTDG1CPK"
        },
        "pk": {
          "S": "account:si-novi"
        },
        "modified_at": {
          "S": "2023-04-24T12:19:07.409Z"
        },
        "slug": {
          "S": "si-novi"
        }
      },
      "SequenceNumber": "000000000000000000935",
      "SizeBytes": 310,
      "StreamViewType": "NEW_IMAGE"
    }
  }
]

Setup

1. Add the dependency to your project

npm install @hicksy/arc-plugin-sandbox-stream

2. Configure your project to use @tables-streams in .arc file

@tables
example
  pk *String
  sk **String

@tables-streams
example

3. Add the plugin to arc config

@plugins
hicksy/arc-plugin-sandbox-stream

4. Setup DynamoDB Local

There are various ways to use DyanmoDB Local. I opted for the bundled version that comes with AWS's NoSQL Workbench tool. But you can use which ever version you're comfortable with.

Just note that if you use the NoSQL Workbench version there's the following caveats I came accross:

Data storage is persistent in the NoSQL Workbench version - I can't find a way to set the inMemory flag available to the standalone / docker version (you'll need to modify any seed scripts to conditionally insert or drop and re-create the table)
You need to be running NoSQL Workbench while you develop and remember to toggle on DynamoDB Local each time (not a biggie really, but it'd be great to have the means to set it to autostart or run as a start-up background task)
The NoSQL Workbench version runs on port 5500, I can't see a way to change this, but it's not a problem for me.

5. Configure Arc to use DynamoDB Local

Arc environment params

Arc makes it really easy to switch out Dynalite for DynamoDB Local.

There's a couple of env vars you need to set to get you going.

Tell arc that you're using an external db
Change the port used for tables within arc
Set your region to ddblocal

You can do this one of several ways.

Either setting an environment variable ARC_DB_EXTERNAL=true / ARC_TABLES_PORT=5500 in an .env file, or using the prefs.arc file and setting the properties within the @sandbox section see arc's docs for more info

e.g.

.env file

APP_URL=http://localhost:3333
ARC_TABLES_PORT=5500

prefs.arc file

@sandbox
external-db true

Arc data seeds

As DynamoDB Local data is persistent when using the bundled NoSQL Workbench version, you'll need to modify any startup data seeds you have and update them to either conditionally insert data, or what I prefer, to delete the table and re-create it each time - this makes it more in keeping with all my other arc projects.

Arc table create or update

Sadly, as Dynalite doesn't support streams, arc's sandbox doesn't create the table with the required StreamSpecification param, so you'll need to supply that either when re-creating the table using createTable or supplying it via an updateTable call.

An example StreamSpecifcation looks like:

StreamSpecification: {
    StreamEnabled: true,
    StreamViewType: 'NEW_IMAGE' 
},

The project I'm working on currently uses Sensedeep's OneTable. So I can achieve the above like:

const client = new Dynamo({client: new DynamoDBClient(params)})
const table = new Table({
    name: DDB_NAME,
    client: client,
    logger: true,
    schema: AppSchema,
    partial: false
})

if(await table.exists()) {
    await table.deleteTable('DeleteTableForever');
}

await table.createTable({
    StreamSpecification: {
        StreamEnabled: true,
        StreamViewType: 'NEW_IMAGE' 
    },
});

You can also do this using the standard JS dynamodb client for any projects that aren't using an additional tool like onetable.

tip: use @architect/functions to infer the full name for your DynamoDB table's, which will be a combination of the name you've given and it's deployement environment etc

6. Modify the polling interval [optional]

By default the plugin will poll for new stream data every 10 seconds. You can override this by providing an alternate millisecond interval by updating your .arc file

@sandbox-table-streams
polling_interval 1000

Get in touch

Any feedback on your usage of this plugin, bugs, or suggestions for improvements please get in touch on GitHub. I'd love to hear from you.

(re-)Introduction

Martin Hicks — Tue, 20 Dec 2022 13:58:47 +0000

Hello 👋🏻

I'm Martin, a UK based Web Developer and AWS consultant.

I've been a member of this community for 4 years now, but purely in read-only mode I'm afraid. Recent events have made me reconsider my online profile and I'm trying to take more ownership of my content.

Still dipping-my-toe into dev.to, but hope to make it a more regular fixture in 2023.

I've just published a handful of back-dated posts from my personal website - martinhicks.net - over here as a starter for ten.

1. A look into a brilliant new Web Component based web framework - Enhance

Martin Hicks

Dec 15 '22

Sharing Enhance elements between projects

#enhance #webcomponents #javascript #progressiveenhancement

Comments

7 min read

2. Some brief thoughts on why I'm excited about a shift towards web-platform first principles

Martin Hicks

Dec 12 '22

Back to the Future

#webplatform #webframeworks #javascript

Comments

4 min read

3. A guide to using 11ty's upcoming 2.0 version along with their new templating language, WebC

Martin Hicks

Dec 17 '22

Eleventy 2.0 & WebC

#webc #11ty #eleventy #staticsite

Comments

11 min read

Looking forward to becoming more active in the community.

Thanks for having me :)

A mistake the internet won't forget easily

Martin Hicks — Mon, 19 Dec 2022 00:00:00 +0000

_ tldr : If you visited my site between 2022-12-01 and 2022-12-18 you might not be seeing updated content on previously accessed pages. I've added a Clear-Site-Data header to this page to possibly rectify. If you don't see multiple blogs posts on the homepage after viewing this article, please manually refresh any page you previously visited - thank you_ 🙏

_ edit: 2022-12-20 - Following a brief discussion with Jake Archibold he offered the suggestion to use a fetch request with the cache: 'reload' property. The nice thing about this is that it's cross-browser, so even Safari on both Desktop and iOS should re-validate their local cache using this technique. It's still not something you'd want to put on every request - I've added an update below that describes this._

A couple of weeks or so ago, enthused by my move to Mastodon, the growing desire of owning your own content post-Musk, and an upcoming New Year's resolution I'd made with myself to be "less mute" on the internet in 2023 - I decided to re-publish my website and start sharing my thoughts in longer form writing.

Over the weekend (hungover, after some pre-Christmas drinks the night before) I realised I'd made a BIG mistake with the website deployment mechanism - accidentally setting a long-term cache-control header for all resources, not just the immutable ones 🤦🏻‍♂️.

Read on for what what caused the problem, how I've resolved it, the steps I've taken to mitigate it, and what I wish was possible to limit damage following a situation like this in the future.

Automatic deployments

I update this site using a GitHub action; triggering a build of my static site and transferring the web assets over to AWS S3 where I statically host my website.

It's within this S3 sync command that I made the galling mistake...

aws s3 sync ./_site s3://martinhicks.net --cache-control max-age=31536000 --delete --acl=public-read --follow-symlinks

If you're not familiar with the AWS S3 cli - this command transfers the directory /_site to the S3 bucket hosting this website.

As you can see I use a number of CLI params:

--delete: This tells the s3 sync command to delete any items that exist in the destination bucket, but aren't in the local source directory (perfect for clearing up old, no-longer needed objects).
--acl=public-read: Every object uploaded to the bucket has public read enabled - this being a public website, you need every asset to be available. Omitting this would prevent you from accessing the page, despite the bucket itself being public
--follow-symlink: I think this is a default actually, but added it in just to make sure any content symlinked into that source directory was actually transferred
--cache-control: The biggie, the doozy.

Here I inform S3 to set the Cache-Control header - a header S3 will include when returning a request for each individual web asset - a html page, a css file, an image, a sitemap.xml... i.e. everything I've just synced.

In haste I inadvertently set the max-age value to 31536000 - that's ONE YEAR - and I told S3 to do so for all my assets. Therefore for every request, S3 will return the header cache-control: max-age=31536000, which tells your web browser that it's allowed to cache the contents of the page you requested for up to one year.

Why is this a problem - caching is good right?

So firstly, caching is great.

It's definitely the right thing to do. You'll improve the speed in which return visitors access your page(s), and you'll receive a better Page Speed Rank from Google, which is a ranking factor on Google's search results.

Using a CDN is even better - I do that here too with AWS CloudFront storing a cache on the edge, closer to where you're accessing the internet from.

eg: you live in London, someone else from London has already accessed this page, you then visit the same page, and the CDN will serve you from the edge cache - win for you as the page will load much faster, bonus for a webmaster as the origin server won't have to handle the request

Anyway that's an aside.

Here I'm taking about the cache-control: max-age=31536000 header I was inadvertently serving.

The issue with that header is the liberal use of it for all my assets. It's absolutely the correct header to return for anything immutable - images, css etc.

If you visit my home page, you'll see that it has some changeable content - a list of latest blog posts that are updated when I publish a new article. So by setting this long-term cache on all web assets, I'd created a situation where the next time you visit my homepage you'd think I hadn't written any new content - you'd likely move on elsewhere - you most certainly wouldn't think;

I know - I'll hit refresh, just in case Martin has messed up his cache headers.

I had an inkling something wasn't right during development, but given I'm an impatient soul, while my site was being synced with S3, in another tab I would be furiously hitting refresh to see the changes

hitting refresh clears the cache, and hid the problem. It was only when I revisited a page on mobile (iOS Safari), that I realised older content was being served.

The correct way to cache content for a website

If you've read this far, you might be interested in seeing a more sensible mechanism for caching content using the S3 sync command.

I updated my GitHub action to make two separate calls to s3 sync.

//long-lived cache for images and css
aws s3 sync ./_site s3://martinhicks.net --cache-control 'max-age=31536000' --delete --acl=public-read --follow-symlinks --exclude '*' --include 'images/*' --include 'css/*'

//no-cache on everything else
aws s3 sync ./_site s3://martinhicks.net --cache-control 'no-cache' --delete --acl=public-read --follow-symlinks --exclude 'images/*' --exclude 'css/*'

The first command ensures that my immutable content (images and css) are served with a year long max-age. Telling the browser that once you've received an image or a css file for the first time to cache it, and keep it in cache for up-to a year. (i.e. your browser won't keep requesting it on subsequent pages, or reloads, over the internet. It'll serve that image from your local browser cache instead.)

The key props to that are as follows:

--exclude '*': tell s3 sync to exclude every item in the source _/site directory (nothing that isn't explicitly included will be transferred to S3)
--include 'images/*' & --include 'css/*: tell s3 sync to include anything within images and the css directory

With the second pass of S3 sync, we invert that - excluding the images and css directories. This run will upload every web asset that isn't in the two excluded directories while setting a cache-control header of no-cache (i.e. don't locally cache the HTML page in browser, re-request it from the CDN or origin as appropriate for every request)

Fixing the problem for returning visitors

Sadly, this isn't a problem that's easily fixed.

First off, you need to hope anyone who accessed page content with a long cache header, will revisit your site to read something new you've written - this is the only way to set a new header after all, as the old content, say the index page, will have already been stored long-term in the returning-visitor's browsers cache.

After studying MDN I found the Clear-Site-Data header, which is exactly what I need.

As MDN states:

The Clear-Site-Data header clears browsing data (cookies, storage, cache) associated with the requesting website. It allows web developers to have more control over the data stored by a client browser for their origins.

Great. So this article is being served with the following header.

Clear-Site-Data: "cache". Which I've acheived by creating a response headers policy on CloudFront that includes this header in the response for this specific path.

Which I sincerely hope means your browser has now been zapped by a Neuralyzer and forgotten any cached data it stored. It's good for all browsers except for Safari (desktop and iOS), and Firefox (although that's behind a feature flag, so hopefully wider roll-out will be in an iminent version).

~(If you're on iOS, maybe you could humour me by giving the homepage and journal page a refresh?)~

Is there a better solution?

I'm currently only applying the Clear-Site-Data header to this page and really hoping / relying on some of the people that read my earlier articles might stumble upon this one? Which is probably unrealistic...

It feels like I shouldn't apply this header on every new page I publish from this point forward - wouldn't that mean the browser will never cache any data? I guess I could include the header for a limited period on all html pages, but it still doesn't feel quite right.

What I really wish is that there was a more deterministic header I could apply, say something like:

Clear-Site-Data: 'cache' '2022-12-01:2022-12-18'

Wouldn't it be handy to be able to say "clear this site cache if the item was stored between the following dates"?

That way I'd be able to drop this header onto all future assets, and eventually anyone returning to my site will have the rogue cached pages cleared - without affecting the behaviour of anything cached before or after the incident date.

Disclaimer: I have no idea how the internals of a browser work. Maybe this is nonsense as the item in cache might just store its expires-at date, rather than the date it was accessed and originally stored?

To be honest, I'm probably over analysing the problem. There's been very little inter-page navigation here - most people have followed a link to a specific article, read it and then moved on. But it's still bugging me...

I'd be keen to hear from anyone who has any better ideas. Is there a more appropriate solution I could apply?

You can contact me at https://indieweb.social/@martinhicks

Update: 20th Dec 2022:

I outlined my issue on the Fediverse - and tagged in Jake Archibald and Alex Russell to see if they had any ideas - I wasn't expecting a response tbh, so was blown away that they took time out to offer their thoughts.

Alex rightly pointed out that Apple lagging behind on this spec (and many more, let's be clear), and the general issues holding up improving the web, are largely down to the lack of browser competition on iOS - something the Open Web Advocacy (OWA) group are looking to change. I completely agree and support this cause, and I'm going to reach out on OWA's Discord to see how I go about helping. If you care about the web, then why don't you too?

Jake offered the following suggestion - fetch(brokenUrl, { cache: 'reload' })

I tested this out on a random test URL and set the long max-time on its cache-control response header. I then hit a second page (after changing the initial test page's cache-control to no-cache), which included the suggested fetch call to the now stuck test page.

After hitting the page with this fetch command, the rogue stuck resource was re-requested by Safari, and when I returned to the original long-cached page it had it's new cache-control: no-cache applied.

It's perfect for what I need - a cross-browser solution, albeit with an additional couple of network requests as you need to specify individual stuck resources - there's no site-wide mechanism like with clear-site-data. But these quick tests proved that you can clear the cache of a previously visited page, from a second independent page (on the same domain).

This technique uses the Request.cache api to tell the browser to re-request the URL specified without using any existing local cache. And to re-populate its cache based on the rules of the new cache-contol header.

Like with clear-site-data you wouldn't want to apply this to all your pages, but if you can get your returning-vistors to view some new content* it's a great solution.

* perhaps a special article such as this, or if you've built a web-app you could email users who you know had accessed your service between a specific date period, and send them to a brand new landing page to peform the cleanse.

There's also an experimental spec proposal only-if-cached on this API too. Meaning in the future, we could get even more specific and only perform the cache reload if the problematic stuck resource is a) in cache for this specific visitor, and b) has a date earlier than the incident.

Anyway, for now.. I've added the following script to this page, and this page only:

<script type="module">

  async function clearRogueCache(url) {
      return await fetch(url, {
          "cache": "reload"
      });
  }

  await clearRogueCache("https://martinhicks.net");
  await clearRogueCache("https://martinhicks.net/articles");

</script>

There we go. Still hoping this article can catch any of the early readers of my site. But also hopeful that if you're reading this way in the future, and you've made a similar mistake, you might have more options at your disposal - lucky you.

Thanks again Jake and Alex - really appreciated.

Eleventy 2.0 & WebC

Martin Hicks — Sat, 17 Dec 2022 17:34:00 +0000

I recently rebuilt this website and my company website - sinovi.uk - using Eleventy 2.0 and their new WebC language for templating.

It's really good. And well worth checking out.

In this post I look at my experiences trying out Eleventy 2.0 and its new Web Component language, WebC.

Note

At the time of writing 11ty 2.0 is pre-release.

It’s just hit version canary-20 and judging by 11ty's recent toot, it’s official release is very close.

Background

I’d heard of the Eleventy project in passing via newsletters and Twitter posts for several years, but up until a few months ago I hadn’t tried to build anything with it.

I think I’d mentally stored it away as similar in vain to Gatsby or some similar react tool - ...there's a lot of those, right?

How wrong was I?

Static site generation

Back in 2018, at Si Novi, we rolled our own internal tool for static site generation, that I’ve now learnt felt pretty similar to 11ty in some respects, but lacked its finesse and feature set.

Our templates were HTML and we used a companion JSON file (ie index.html & index.json) to hook data into each page view using HTML attributes.

We had collections in an array within a standalone json file (eg articles.json). We even hooked it up to PHP blade templates for one test project.

There was a key thing wrong with it though - it was a bloody pain to use.

At the time we didn’t know about the front matter syntax, which in hindsight might’ve helped.

But the key issue was our node CLI processing engine was written using a jquery compatible dom lib - Cheerio, and a load of Grunt hooks.

It was fiddly; hard to maintain and lacked features, and we never quite found the time to make improvements alongside our client work.

Furthermore, we were having to stuff lots of properties into the JSON files to handle conditional templating logic particularly for reusing partial views / re-usable template snippets.

Maybe if we’d known about frontmatter, had chosen a templating engine like nunjucks, and had known about parse5, I’d still be maintaining it now - who knows?

What I do know (now) is that 11ty absolutely nails static site generation.

They have multiple templating languages, nested layouts, a sensible config and plugin system, and a really cool data cascade which provides lots of options for populating a page’s templating data or mutating a particular value prior to generating the page.

The docs site is veeeeerrrry comprehensive. To be honest… so big I found it overwhelming to begin with (it really melted my head for a bit), but it's a fantastic resource and a credit to the community of contributors.

My first look

I took my first spin of Eleventy when I heard about Enhance in September and noticed one of their deployment targets was 11ty.

Building out a few demo pages, I really liked what I saw in both projects but struggled with a few bits as I tried to understand how these worked together.

Attempting to learn two new things at once meant I was doing a disservice to both tools; hampering my understanding of each project, and limiting my discovery of any overlap / boundaries of concern when using them together.

I gave up.

But promised myself I’d go deeper into each tool individually when time allowed.

Learning Eleventy

A month or so later I started a fresh starter Eleventy project.

Making some really simple pages and layouts, I learnt about 11ty’s special dirs (like _includes & _data), and generally started to feel more comfortable with how Eleventy worked and it’s limitations.

Those limitations, for me, were the fact that sharing blocks of re-usable components felt difficult.

You could use 11ty’s shortcodes to create snippets, or nunjuck macros, but these felt to me like reusing strings of templates was just about ok, but making them configurable using properties or attributes wasn’t a great experience for me.

Having first used 11ty with Enhance, I’d been drawn to Enhance’s use of Web Components as reusable elements - it fit my more recent mental model of using components in htm or react.

Maybe I gave up too soon in my earlier experiments?

But then I heard about Webc - I decided to persevere with my original plan, and set about re-building the Si Novi and martinhicks.net sites .

Webc - now we’re rocking

Webc is a brand new 11ty templating language.

It uses Web Components and requires Eleventy 2.0 (at time of writing this is pre-release and requires installing their canary package).

Using Webc with 11ty is seamless; You can use WebC to build an individual component, for re-use and you can use it for layouts. Basically anywhere you’d expect an 11ty template to work, .webc files work.

Meaning it works on its own, or can be used within their other templating languages too. And being 11ty you can mix and match templating languages throughout a project.

I like this 11ty feature a lot - there’s loads of useful 11ty code snippets on gist, on their website, and, well all over the internet written in Nunjucks or Liquid and choosing to use WebC doesn’t prohibit you from tapping into this rich seam.

For example I’m using an njk page to create my sitemap.xml and another to create my RSS feed.xml.

//sitemap.njk

---
permalink: /sitemap.xml
eleventyExcludeFromCollections: true
---
<?xml version="1.0" encoding="utf-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    {% for page in collections.all %}
        <url>
            <loc>{{ site.url }}{{ page.url | url | replace(r/\/$/, "") }}</loc>
            <lastmod>{{ page.date.toISOString() }}</lastmod>
        </url>
    {% endfor %}
</urlset>

//rss.njk
---json
{
  "permalink": "feed.xml",
  "eleventyExcludeFromCollections": true,
  "metadata": {
    "title": "Martin Hicks - Journal",
    "subtitle": "Martin Hicks is a software developer from Manchester, UK",
    "language": "en",
    "url": "https://martinhicks.net/",
    "author": {
      "name": "Martin Hikcs",
      "email": "hello@martinhicks.net"
    }
  }
}
---
<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xml:base="{{ metadata.url }}" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>{{ metadata.title }}</title>
    <link>{{ metadata.url }}</link>
    <atom:link href="{{ permalink | absoluteUrl(metadata.url) }}" rel="self" type="application/rss+xml" />
    <description>{{ metadata.subtitle }}</description>
    <language>{{ metadata.language }}</language>
    {%- for post in collections.articles | reverse %}
    {%- set absolutePostUrl = post.url | absoluteUrl(metadata.url) %}
    <item>
      <title>{{ post.data.title }}</title>
      <link>{{ absolutePostUrl }}</link>
      <description>{{ post.data.description | htmlToAbsoluteUrls(absolutePostUrl) }}</description>
      <pubDate>{{ post.date | dateToRfc822 }}</pubDate>
      <dc:creator>{{ metadata.author.name }}</dc:creator>
      <guid>{{ absolutePostUrl }}</guid>
    </item>
    {%- endfor %}
  </channel>
</rss>

I also really like the fact I can just use WebC as a templating language for HTML generation.

This website currently has zero need for any client-side progressive enhancement, so I don’t need any of the web component features in the HTML served to the browser.

11ty allows you to build a component that just returns HTML, and if so, the generator treats that component output as just the inner HTML omitting the enclosing custom element tag.

If you did want to keep the component wrapper for some reason you pass an attribute of webc:keep.

//my-avatar.webc
//example html only web component
<picture>
    <source srcset="/images/5F8AB69C-FA08-4C25-B932-74D76EBB7721.webp" type="image/webp">
    <img src="/images/5F8AB69C-FA08-4C25-B932-74D76EBB7721.jpg" alt="Me and my wife, Helen, lying on the grass in summer. " :width="this.width" :height="this.height" class="max-w-[100px] md:max-w-[200px] mx-auto aspect-square ring-2 ring-zinc-500/40 rotate-45 rounded-full bg-zinc-100 object-cover"/>
</picture>

Webc with 2.0 means I could now build proper re-usable components, configurable with attribute props if required. No more snippets or Nunjucks macros.

<my-avatar width="200" height="200"></my-avatar>
<my-avatar width="100" height="100"></my-avatar>

Perfect.

Other wins are;

Using web components within the head element - this isn’t allowed by the Web Component spec I don't think, but given you may have a Webc component that just provides HTML you can use the webc:is attribute to upgrade a standard element to a WebC component just for templating purposes (but only if it just returns html)

Eg:

<script webc:is="json-ld" ></script>

The above script tag in my head, will be ran using my component json-ld, which basically adds some dynamic json-ld for articles on this site, and excludes if it's not an article page.

WebC components can include render only Js functions to iterate collections

---
meta:
title: "Articles"
description: "Occasional thoughts"
pagination:
  data: collections.articles
  size: 10
  alias: articles
  reverse: true
layout: layouts/main.webc
---


<container>
  <div class="flex flex-col mx-auto justify-center ">
    <h1 class="text-4xl font-bold tracking-tight text-zinc-800  sm:text-5xl mt-4">
      Journal
    </h1>

    <div class="grid grid-cols-1 gap-8 md:grid-cols-2 auto-cols-auto md:auto-rows-[1fr] mb-8">
      <script webc:type="render" webc:is="template">
        function () {
          //console.log(this.pagination)
          let articles = this.pagination.items;

          return articles.map((article, idx) => /*html*/`

            <div class="prose relative pb-8">
                    <a href="${article.data.url}">
                        <picture class="flex w-full " >
                            <source srcset="${article.data.image.webp}" type="image/webp">
                            <img class=" full-width mb-2" src="${article.data.image.path}" width="345" height="236" alt="${article.data.image.alt}">
                        </picture> 
                    </a>
                    <span class="!text-sm">
                    ${article.data.date}
                    </span>
                    <h1 class="!text-xl mb-2 font-semibold"><a class=" no-underline" href="${article.data.url}">${article.data.title}</a></h1>
                    <p>${article.data.description}</p>
                    <a class="absolute bottom-2 " href="${article.data.url}">
                        Read the article
                    </a>
                </div>

                    `)
          .join("");
        }
      </script>
    </div>

    <hr>

  </div>

  <my-details mode="full"></my-details>
</container>

Slots

If you’ve used any Web Component tool or manually created your own, you’ll know that web components use ‘slots’ to control where nested elements or strings are displayed within the component template.

They’re super useful and help direct content to the correct placeholder without using attributes or similar.

eg:

//social-link.webc

<a class="group -m-1 p-1" :href="href" target="_blank" :aria-label="this.arialabel" :role="this.role" :rel="this.rel">
    <div class="flex items-center space-x-2">
        <slot name="icon"></slot>
        <slot name="content"></slot>
    </div>
</a>

Which is usable like:

<social-link rel="me" role="listitem" href="https://indieweb.social/@martinhicks">
    <icon-mastodon slot="icon" class="w-8 h-8"></icon-mastodon>
    <span slot="content">Follow on Mastodon</span>
</social-link>

nb icon-mastodon is another webc component - completely nest-able as you'd expect

Things to look out for in 2.0 / webc

Having not been a long time user of 11ty, I’ll leave any deep comparison between the two versions to seasoned experts.

There’s loads of new features in 11ty 2.0, some of which are breaking changes.

Their docs site does a good job of signposting these changes, and I’m sure when it’s released there will be loads written to guide users in migration.

What I’ve found:

Webc: Script and link tags

Since, I think, "@11ty/eleventy": "2.0.0-canary.18" or "@11ty/eleventy-plugin-webc": "0.8.0", you’ve been required to add webc:keep to any script or link tag that has an external src.

Thankfully the CLI warns you of these during build, throwing an error.

However, I found that several non external script tags in my head (two json-ld and the local google analytics gtag script) weren’t included in my published site for a week.

Adding webc:keep brought them back.

Maybe I misread the CLI warning, but I don’t think I did. And certainly the lack of them on an external src caused a build fail, whereas omitting them for locally src’d elements didn’t.

Luckily I don’t care much about either of those on my personal site, so no biggie.

__**Given that I’m using a canary build and I haven’t been studiously keeping up with the change notes between pre-release build versions I’ll take the blame on this one.**

11ty 2.0 - The copy command doesn’t actually copy files locally during dev

This is new and intentional, for performance reasons, it sort of magically symlinks them internally during the local serve process.

So as far as the browser is concerned the images folder you’ve set to copy to the output dir, for example, and therefore is served from /images/myimage.jpg, hasn’t physically been copied to that location on your machine. That only happens during a production build.

Fine when you know but it’s a little confusing at first.

I’ve had to build a few production builds locally at times just to make sure my copy configs are set correctly.

Plug-ins have a whole bunch of new hooks

This is a big upgrade and I think will make integrating other tools way easier.

I think it's backwards compatible. I’m looking forward to playing around with this more.

WebC components aren’t automatically discoverable within a project by default.

I found this confusing.

Especially as most of my .webc components were simply returning HTML, I didn’t want to have to add a load of webc:import attributes per component.

Thankfully 11ty’s config system has you covered.

Adding the following to your .eleventy.js file, and placing all your components within /_includes/components makes them usable throughout the entire project without individually importing them.

eleventyConfig.addPlugin(pluginWebc, {
    // Glob to find no-import global components
        components: "src/_includes/components/**/*.webc",
});

Everything in project root by default

I think this is right, and not just a mistake I made. But the default 11ty starter had everything configured to run from the project root.

Eg, index.webc was in the same root folder as package.json and other non web assets.

I didn’t like this.

Again, 11ty config to the rescue, it’s super simple to tell 11ty where your input and output dirs should be. So it was quick get things how I wanted them.

return {
  dir: {
      input: "src",
      output: "_site"
    }
  }

It’d be great if the starter cli command could ask you where you’d like your source and output directories to be, and auto configure this for newbies like me.

Tailwind

Tailwind works really well with a component based system, so 11ty and webc is no different, outputting only the css for classes that you’ve actually used in your pages.

To make that work better, I pointed tailwind config to the output folder (_site in my case) to look for content, rather than the src directory, and made it run after 11ty prod build.

//tailwind.config.js
module.exports = {
  content: ['./_site/**/*.html'],
  plugins: [require('@tailwindcss/typography')]
}

This way any draft components I’ve built but not yet used, won’t have any of its unique classes included in the final production css output. Until they’re actually included in one of the html pages.

I also needed a way to allow component modification throughout the site (more so on Si Novi). To do that I used the Tailwind Merge package and created a helper function within .eleventy.js that can be used on each component.

eleventyConfig.addFilter("tailwindMerge", function(defaultClasses, overrideClasses) { 
  return twMerge(defaultClasses, overrideClasses)
});

This means I can pass in overriding css attributes at the point of using the component and the twMerge function (as it’s tailwind aware), replaces the defaults with the overrides.

Within a WebC component:

//link-primary.webc
<a :href="href" :class="tailwindMerge('underline hover:text-blue-500', this.class)"><slot></slot></a>

If I used this component, like so...

<link-primary class="hover:text-red-500">example link hovers red</link-primary>

...the hover on that instance of the component would be red.

Wrap up

I’m so glad I’ve found 11ty, it’s a great tool. Version 2.0 and WebC has made it sticky for me.

I haven’t used any of the more advanced webc features such as scoped css or bundling, but I’m sure I’ll test them out in due course.

Absolutely great job.

Keep an eye out for the next post in this series, which explains how to set up 11ty, AWS and GitHub actions to create a CI/CD build pipeline to deploy to S3.

You can view the source code of my site here.

Sharing Enhance elements between projects

Martin Hicks — Thu, 15 Dec 2022 17:00:00 +0000

This post introduces a new arc plugin for Enhance projects that I've recently published, with the catchy name shared-enhance-components-plugin.

Read on for details about how to use the plugin and why I came to write it.

Background

While I continued my exploration of the Enhance framework, I started to wonder how you'd go about importing elements from a shared library or UI toolkit.

In a React project, you'd import the library into the given component and then begin using its JSX tag within the render.

However, Enhance elements are dependency free by design, for good reasons.

So instead, any element you create within /app/elements or define within /app/elements.mjs (* more on this later), becomes automatically available in any Enhance page.

That doesn't mean you can't use dependencies where appropriate. Let's say you have a couple of elements, one of which can be used standalone, the other you always want a particular element included, you could do the following:

//app/elements/csrf-input

export default function CsrfInput({ html, state }) {
    const { attrs={}, store={} } = state
    const { name = 'csrf' } = attrs
    return html`
        <input type="hidden" name="${name}" value="${store.csrf_token}" />
    `
}

import CsrfInput from "./csrf-input.mjs";

export default function CsrfForm({ html, state }) {
    const { attrs={} } = state
    const { action = '', method = '' } = attrs

    return html`
    <form action="${action}" method="${method}">
        ${CsrfInput({html, state})}
        <slot></slot>
    </form>
`
}

You can simply call the pure function to return its "render", e.g. ${CsrfInput({html, state})}.

And I think looking at my browser's source, it means the dependent element is just the HTML entity, rather than being expanded into a web component. Which in this case is what I wanted, all be it by accident.

Pretty cool.

But what about if you want to bring in a shared UI library maintained by a different team or 3rd party?

After a search on the Enhance Discord (which is great by the way - definitely head there for help / guidance / cool things people are building), I couldn't find anything baked in.

I did see a comment from Macdonst (one of the begin/enhance team), who was explaining how you could do all of your imports in an /app/elements.mjs file, and then they would be available throughout your app just like a first-party element you've created in /app/elements - nice.

He went on...

The above is begging to be a plug-in.

So I thought why not.

What is this elements.mjs file anyway?

It's not mentioned much in their docs at all. In fact, at the time of writing, it's only referenced within the Deploy to Fastify section.

However, if you look at the code for the arc-plugin-enhance (which is the plugin that orchestrates arc to have a catch-all lambda for each /app/api/ route, among many other things), you'll see that 4 locations are checked for elements to enhance:

let pathToModule = join(basePath, 'elements.mjs')
let pathToPages = join(basePath, 'pages')
let pathToElements = join(basePath, 'elements')
let pathToHead = join(basePath, 'head.mjs')

i.e.

/app/elements.mjs
/app/pages (any mjs file in here)
/app/elements (any mjs file in here)
/app/head.mjs

The elements.mjs file is basically returning a keyed object mapping the tag name (e.g my-component) to a corresponding element function (e.g. MyComponent) (which can be located anywhere, like node_modules/some-package/elements/MyComponent.mjs)

import MyComponent from 'some-package/elements/MyComponent.mjs'

let elements = {

  'my-component': MyComponent
}

export default elements

So that's a pretty powerful in-built mechanism for linking things together right there.

The need for a potential plugin is that you may npm install a package with 10s, 100s of components you'd like to use, so having a plugin do most of the leg work is helpful.

Also, defining a map between the tag name and pure function doesn't seem to have any draw back - I'm fairly certain elements are only processed if they're referenced within a page and / or element - so mapping a lot of potentially unused components seems to have no issues

Building the plugin

Enhance (the way I'm using it), is wrapped with Architect. Meaning you can easily deploy your enhance app to AWS, or to Begin.

Because it uses Architect's sandbox to run locally, and its hydration mechanism to hydrate your Lambda function code before deployment, you can also tap into Arc's powerful plug-in system.

I'd previously had a go at an Architect plugin and got a little lost. Since then (18 months or so ago), they've massively expanded their plug-in docs and improved the available lifecycle hooks you can tap into.

I also had a good look at some of the plugins listed in their repository, which helped me figure things out.

After an afternoon or so's work I had an acceptable (just about) version of a plugin which works as follows:

You add a component package or UI library to the .arc file under the plugin's unique @ pragma - @shared-enhance-components-plugin - this informs the plugin which external packages to import component elements from.
On arc hydration (which happens during sandbox start, and pre arc deploy), the plugin analyses the available functions that can be imported from the given package:

- either from a specific folder, in which case each .mjs / js file will be mapped into elements.mjs
- or from an index.js file, in which case each named export will be mapped into elements.mjs

Tag names are inferred by de-camelcasing the function name
Any existing elements you've manually added to elements.mjs will be preserved (auto generated lines include an eol comment)

Example usage

I think it's pretty easy to use.

First install the plugin and tie it into your project's arc file.

npm install @hicksy/shared-enhance-components-plugin

//.arc

@app
myproj

@plugins
enhance/arc-plugin-enhance
hicksy/shared-enhance-components-plugin

nb: you have to drop the preceding @ on scoped package names in the .arc file so they don't collide with the pragma names

Install some shared elements

For example, we could install these example Enhance form elements - https://github.com/enhance-dev/form-elements.git

npm install git+https://github.com/enhance-dev/form-elements.git

Tell the plugin to use this package

//.arc

@app
myproj

@plugins
enhance/arc-plugin-enhance
hicksy/shared-enhance-components-plugin

@shared-enhance-components-plugin
hicksy/enhance-csrf 'elements'
enhance/form-elements

Note in the above example the package we've just installed is referenced just by it's name (dropping the @ again to avoid collision with arc pragmas) - we know by looking at this package all of it's components are named exports from a route index.js file.

For @hicksy/enhance-csrf package, we pass a second arg to the plugin, this tells the plugin that the components are individual files within the specific folder.

Start

npm start

The plugin-hook will fire, and the /app/elements.mjs file will be auto-populated with the tag-name's and import declarations as required.

// /app/elements.mjs

import CsrfInput from '@hicksy/enhance-csrf/elements/csrf-input.mjs' //automatically inserted by shared-enhance-components-plugin
import CsrfForm from '@hicksy/enhance-csrf/elements/csrf-form.mjs' //automatically inserted by shared-enhance-components-plugin
import { CheckBox } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { FieldSet } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { FormElement } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { LinkElement } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { PageContainer } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { SubmitButton } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin
import { TextInput } from '@enhance/form-elements/index.js' //automatically inserted by shared-enhance-components-plugin

let elements = {

  'csrf-input': CsrfInput, //automatically inserted by shared-enhance-components-plugin
  'csrf-form': CsrfForm, //automatically inserted by shared-enhance-components-plugin
  'check-box': CheckBox, //automatically inserted by shared-enhance-components-plugin
  'field-set': FieldSet, //automatically inserted by shared-enhance-components-plugin
  'form-element': FormElement, //automatically inserted by shared-enhance-components-plugin
  'link-element': LinkElement, //automatically inserted by shared-enhance-components-plugin
  'page-container': PageContainer, //automatically inserted by shared-enhance-components-plugin
  'submit-button': SubmitButton, //automatically inserted by shared-enhance-components-plugin
  'text-input': TextInput //automatically inserted by shared-enhance-components-plugin
}

export default elements

Publishing to NPM

I've never published a package to NPM before so this was all new to me.

Previous node packages I've created have all been private and we'd opted to npm installing from the git repo address rather than purchasing a private packages account from npm.

Publishing was really straight forward (aside from accidentally deploying it unscoped, and then deciding I'd prefer to release it scoped to my username).

I followed this guide to npm publishing - thanks Zell!

And there you go...

A small plugin that hopefully eases the use of shared Enhance elements and hopefully making it more compelling for authors to share libraries of their elements for others to use?

There's no doubt a few wrinkles will be discovered, and the plugin code could do with a tidy, but I'm pretty happy with it.

Get in touch on GitHub if there's any issues, improvements or feature ideas.

Back to the Future

Martin Hicks — Mon, 12 Dec 2022 17:12:00 +0000

As 2022 draws to a close I wanted to reflect on a welcome change in the web framework world that shifts the needle back towards web-first principles. Which I'm all here for.

This year I’ve worked on a large web application using Remix, and the past week I’ve finally had free time to properly explore Enhance - something I've been super excited about since it was announced back in late Summer 2022.

Both frameworks share core principles;

web-platform aligned mechanism for storing state (i.e forms, sessions and http verbs - those foundational web principles that somehow had been forgotten)
progressive enhancement; a web app can (and should) be usable without browser JS
HTML served over-the-wire
file based routing

The fact as an industry we allowed ourselves to sleep-walk into a period where these principles weren't the norm is unbelievable.

Thankfully a long overdue course correction is taking place.

Remix

I really enjoyed building with Remix - it's well thought out, and makes working with React, well... more enjoyable.

We were able to scaffold out a broad approach to the app pretty seamlessly, and the build out was relatively pain free. We deployed the app to AWS using what Remix call their Grunge stack - which basically pairs Remix with Architect (a tool I'm already very familiar with) and provides a catch-all Lambda to power the server-side code.

I believe Remix can be deployed anywhere that supports the Node.js runtime, and I think it even runs at the edge on web-workers - which is pretty cool.

It's amazing how much cruft Remix helps take away from a typical React project- no more nonsense client side Redux stores, optimistic UI is extremely easy (no loading spinners), and with the file based routing it's far simpler to move things around as you're building out an app.

It's not without complexity, it's still React, right? You have to be pretty disciplined to not over engineer things, particularly the functional component, as you can get drawn to using (or overusing in my case) hooks etc to maintain state client-side rather than using the platform to best effect. (Aware this is my issue not necessarily Remix's).

A lot to like and definitely looking forward to using it again in 2023.

Enhance

For those new to it, Enhance is a standards based web framework underpinned by Web Components from the folks at Begin.

So far I've only had a chance to test out some very basic examples, but really like what I've seen, both in terms of playing around with it and the team's core mission.

Our mission is to enable anyone to build multi-page dynamic web apps while staying as close to the platform as possible.

YES!! 🙌

I'll write up another post soon I'm sure, but here's a quick summary of wins:

1. No need for client-side code

Components (or Elements as enhance refers to them), can be just HTML if you like, there's no need for any client-side JS at all.

export default function ProseHeader({ html, state }) {
  const { attrs } = state
  const { title = '' } = attrs

  return html`
    <h1><slot></slot></h1>
  `
}

This Element would be usable in any Enhance page like so:

<prose-header>Article title</prose-header>

And to the browser, would be served up expanded ready for enhancement if required:

<prose-header>
<h1>Article title</h1>
</prose-header>

One thing I wish were possible (and it may well be, I just haven't looked hard enough), is whether you could opt-out of the web component parent element all together for elements that don't require client-side enhancement
i.e. they only contain HTML.

This is a feature I quite like in 11ty's webc - meaning if you're using webc as only a templating language, the outputted HTML is just pure HTML.

Like I say, I'm not sure if this is possible or not. Nor is it much of a pain point to be honest, just a nicety.

2. Progressive enhancement

Easy to progressively enhance client-side capabilities by adding a script into the string returned from the Element.

...rest of Element
<script type="module">
  class ProseHeader extends HTMLElement {
    constructor() {
      super()
      this.heading = this.querySelector('h1')
    }

    ...whatever powers you want to give the new Web Component
  }

  customElements.define('prose-header', ProseHeader)
</script>
...etc

3. Routing

File based routing, with an API mechanism backed in.

For example, let's say you have the following page /app/pages/index.html, and the following api route /app/api/index.mjs the returned JSON from either the get handler, or post handler of the API will be made available to any custom elements within index.html automatically.

The server-side mechanism automatically routes to the get handler if the request method is GET and likewise to the post handler if the request is a POST.

This is similar in nature to Remix's actions (post) and loaders (gets), but split across different files.

4. Automatic state

Server side state is populated into every custom element for a given route automatically

I think this is really powerful. No need to drill props, no need to maintain some sort of client-side state, if it's returned in the json prop from either a get or post handler it will be available in state.store in every element on the current page.

What a joy.

Summary

Overall two very interesting frameworks that I'm delighted exist.

They've both, for different reasons, helped me get excited about web development again and travelling back to the future.

DEV Community: Martin Hicks

Introducing Dynoxide: a fast, embeddable DynamoDB engine

From library to tool

Under the hood

Get started

Using Whisper and FFmpeg to build a fully automated YouTube Shorts pipeline

The manual era

JSON and Python: the first automation

Automating everything else

Fixtures from the FPL API

Video transcription with Whisper

Results from the API

Encoding publish-ready Shorts

Upload to YouTube

The weekly routine from Gameweek 32

What I'm proud of

Recently Played: bringing back my Last.fm component

The architecture

Data layer: src/_data/listening.js

The component: now-listening.webc

Server-rendered HTML

A <template> element

Client-side polling script

Design decisions

Was it worth it?

Remove trailing slash on Eleventy S3 hosted sites using Cloudfront function

Intro

Background

Solution

1. Create a new CloudFront function

2. Publish the function

3. Configure your Cloudfront distribution to route requests through the function

Wrap up

Enhance CSRF package - now supports multipart form data

Get in touch

DynamoDB Streams locally with arc.codes & Enhance

Introduction

Background

Using the plugin

Setup

Arc environment params

Arc data seeds

Arc table create or update

Get in touch

(re-)Introduction

Sharing Enhance elements between projects

Back to the Future

Eleventy 2.0 & WebC

A mistake the internet won't forget easily

Automatic deployments

Why is this a problem - caching is good right?

The correct way to cache content for a website

Fixing the problem for returning visitors

Is there a better solution?

Update: 20th Dec 2022:

Eleventy 2.0 & WebC

Background

Static site generation

My first look

Learning Eleventy

Webc - now we’re rocking

Things to look out for in 2.0 / webc

Wrap up

Sharing Enhance elements between projects

Background

What is this elements.mjs file anyway?

Building the plugin

Example usage

Publishing to NPM

And there you go...

Back to the Future

Remix

Enhance

1. No need for client-side code

2. Progressive enhancement

3. Routing

4. Automatic state

Summary

Data layer: `src/_data/listening.js`

The component: `now-listening.webc`

A `<template>` element