DEV Community: Sid Vishnoi

Spec Prod: a GitHub Action for validating and deploying W3C specs

Sid Vishnoi — Thu, 17 Sep 2020 14:17:05 +0000

My Workflow

Action name: Spec Prod

tl;dr: I created a GitHub action that simplifies the compilation, validation and deployment of specifications in the W3C ecosystem.

This post is solely for the purpose of submission. Read more about the action and its development in my other blog post: Simplifying W3C standards workflow with a GitHub Action.

Submission Category:

DIY Deployments

Yaml File or Link to Code

w3c / spec-prod

GitHub Action to build ReSpec/Bikeshed specs, validate output and publish to GitHub pages or W3C

Spec Prod | Documentation 📘

This GitHub Action lets you:

Build ReSpec and Bikeshed specs.
Validate generated document's markup and check for broken hyperlinks.
Publish generated spec to GitHub Pages and/or w3.org (using Echidna).

Basic Usage

During a pull request, the action:

figures out if you're using ReSpec (index.html) or Bikeshed (index.bs)
converts the ReSpec/Bikeshed source document to regular HTML
runs broken hyperlink checker, and validate markup using W3C nu validator

Additionally, if a commit is pushed to the "main" branch, the action deploys the built specification to /TR/.

# .github/workflows/auto-publish.yml
name: CI
on
  pull_request: {}
  push:
    branches: [main]
jobs:
  main:
    name: Build, Validate and Deploy
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v4
      - uses: w3c/spec-prod@v2
        with:
          W3C_ECHIDNA_TOKEN: ${{ secrets.ECHIDNA_TOKEN }}
          # Replace following with appropriate value. See options.md for details.
          W3C_WG_DECISION_URL:

…

View on GitHub

Additional Resources / Info

Projects using this action

https://github.com/w3c/gamepad/
https://github.com/w3c/web-share/
and more (hopefully all W3C specs 🤞) coming soon!

Simplifying W3C standards workflow with a GitHub Action

Sid Vishnoi — Mon, 14 Sep 2020 18:22:23 +0000

What is it about? In the W3C world, ReSpec and Bikeshed are two common tools being used to write web standards. ReSpec is a JavaScript based tool that runs in browser and enhances a HTML document to generate a static HTML document that is suitable to be published as a standard. On the other hand, Bikeshed is a Python based command line tool which processes a superset of markdown to create similar output. ReSpec also has a CLI. Further, both tools are generic enough to be used (and are being used) outside the W3C. I wrote a bit more about ReSpec in another post about the ReSpec MOSS Grant.

Why a compile step is needed? ReSpec enhances a HTML document in the browser on runtime, so it is simpler to use (no special tools or dependencies required) and a ReSpec document can be directly hosted on GitHub pages. Bikeshed uses a special text format, so it cannot be hosted directly without a compile step. But as ReSpec does everything at runtime, even with great caching practices, a ReSpec document is slower to load. So, people sometimes use the CLI to compile a ReSpec document and create a static HTML output, as Bikeshed does, and deploy it to GitHub pages.

Why an Action is needed? The W3C standards ecosystem is large, and many repositories have setup their own workflows using Travis, GitHub Actions etc. to build, validate and deploy their specs to GitHub Pages or W3C. This leads to fragmentation and it becomes difficult to maintain all these workflows. So, I decided to create a GitHub action, called Spec Prod, that can be used by all the specs with minimal effort.

What does the action do?

When setup in a workflow, the action figures out whether the repository uses Bikeshed or ReSpec, and uses appropriate tools to generate a plain HTML document.

Then it runs various validators, like the W3C markup validator and checks for broken hyperlinks. This serves as a check so we don't accept malformed PRs.

When a pull request is merged, the action can deploy the generated HTML document to GitHub pages. When appropriate, it can also deploy the specification to the W3C website. You can look at a sample run.

Developing the action

In this section, I'll share how I developed the action and I hope you can take away a things or two from my experience!

A composite action

GitHub actions recently started to support "composite" actions, so I wanted to try that. In plain words, you can write multiple small programs/scripts and run them one after other as you run "steps" in a workflow. The composite actions have some work to do on GitHub side, like UI improvements and conditional step execution, but it worked well enough for my use case, with some "hacks".

For the separating steps in the action (not workflow), I decided to use the group command. So, all the steps in my action are wrapped in a collapsible section:

# action.yml
- run: |
    echo "::group::Step 1"
    run-script-for-step-1
    echo "::endgroup::"
  shell: bash
- run: |
    echo "::group::Step 2"
    run-script-for-step-1
    echo "::endgroup::"
  shell: bash

For conditional execution, the scripts do a check and exit(0) if that step doesn't need to run. See example.

It's a bit ugly, but it works.

Choosing the right language

I love JavaScript. But for this action, I decided to write it in Bash (kinda insane, but I love bash too). Turns out, bash gets out of hands really easily, specially when it comes to string processing and error handling. So, after finishing the "mostly working" action, I decided to re-write it in JavaScript. Some part is still in Bash, perhaps for good ol' memories. Bash is actually great for running sequential commands.

Apart from the translation, it didn't require much effort in the action. I had to mostly change the calls like:

# action.yml
- ${{ github.action_path }}/build.sh
+ node ${{ github.action_path }}/build.sh

Choosing good default inputs

A good GitHub action (or any tool in general) should have a good default behavior. Even when you do not pass any inputs to the checkout action, it does what you expect it to do - it checks out a branch/ref. A good tool should have a good default behavior, and a low learning curve.

The Spec Prod action, also does not require any input. By default, it builds and validates the document. Want to deploy to the GitHub pages branch also? Specify the branch name. Want to deploy to W3C? Authenticate. Should it deploy on PRs? Not likely, so disable all deployments on PRs, unless asked.

Allow overrides

The default behaviour is not always desirable and a software should be flexible enough when needed. So, the Spec Prod action also allows users to change the behavior. Don't want to run a validator? Tell it not to. The input file does not have a generic name? Specify it.

Add a pre-process step

As a multi-step action, it helps if you can pre-process all the inputs at once, and pass them to other steps in a "standardized manner". This advice also works in other cases, as long as you don't hit separation of concerns issues.

In a GitHub action, as all inputs are key-value pairs, and it's not cool to ask users to input a stringified JSON value, I decided to prefix input keys with their categories, and pass validated and structured JSON as inputs to internal steps. So, the plain key-value inputs:

- uses: sidvishnoi/spec-prod@v1
  with:
    W3C_ECHIDNA_TOKEN: SECRET
    W3C_MANIFEST_URL: URL
    W3C_WG_DECISION_URL: ANOTHER_URL

become:

const inputs = {
  // ...
  deploy: {
    w3c: {
      token: "SECRET",
      manifest: "URL",
      decisionUrl: "ANOTHER_URL",
    },
  },
};

and we just pass the required inputs a step:

- run: node deploy-w3c.js
  env:
    INPUTS_DEPLOY: ${{ toJson(fromJson(steps.prepare.outputs.all).deploy.w3c) }}

Also, input names should be descriptive enough that users don't always need to read the full documentation to understand what's going on. So, instead of W3C_TOKEN, I used W3C_ECHIDNA_TOKEN, as it makes clearer (to the W3C folks) that the token is obtained through Echidna.

Keeping it simple - No dependencies

Personally, I don't like build-steps unless required. I like TypeScript, but I didn't write this action in TypeScript because of an seemingly unnecessary (hot take) build step. Instead, I used the // @ts-check comment and JSDoc annotations to type-check/hint code within the editor. It works pretty well.

Also, I didn't use the Actions Toolkit, or any other node_modules, and decided to create my own as needed. It's not super productive, but good to have some fun. You don't always need npm; copy-paste works well enough sometimes: compare this and this. Also, in my defense, I originally wrote this entire action in Bash, where npm isn't a thing.

Deploying to GitHub pages

If you've used GitHub Actions to deploy to GitHub Pages, you must have come across peaceiris/actions-gh-pages. It's a great action, and does the job.

But there is a catch. GitHub composite actions do not yet allow you to run another GitHub action as a step. It'll be supported in near future, and that would open a lot of opportunities! Also, it is not always super-safe to trust code by others, specially when you pass it a GitHub access token.

I needed to support GitHub pages deployment regardless of that support. So, I decided to rewrite it myself. It took me around 60 lines to Bash to come up with a well working solution. And then later, a near 100 line rewrite in JS, including more validations and logging. Feel free to copy-paste. Feedback welcome!

Testing

Testing is presently the weakest part of GitHub actions. Other than simple unit tests, the only way to test the action is to push it, and send PRs/commits to a test repository. That also means a lot of test runs with commits like: "please work, does it work now?, done, looks done now, why don't you work!" etc.

For some parts, I created a local environment to run the steps with test inputs, but it's still cumbersome.

What's next?

I will be sending PRs to the repositories that can make use of this action. I've got plenty of open issues to work when I get free time. Contributions are welcome!

w3c / spec-prod

GitHub Action to build ReSpec/Bikeshed specs, validate output and publish to GitHub pages or W3C

Mozilla Open Source Support Grant — Helping Make Open Source Sustainable

Sid Vishnoi — Fri, 24 Jul 2020 13:08:31 +0000

I was awarded with a Mozilla Open Source Support (MOSS) grant through which I was funded to work full-time on an open-source project (ReSpec) for over 3 months. In this post, I’ll share my experience.

What is MOSS?

The MOSS awards program financially supports open source projects that contribute to Mozilla’s work and the health of the Internet. Specifically for the MOSS Foundational Technology track, Mozilla looks for projects that Mozilla products or Mozzilians rely on.

What is ReSpec?

ReSpec is an open source project that makes writing international standards easier. For specification authors, ReSpec handles mundane tasks like automatically generating legal and conformance boilerplates, linking citations, generating a bibliography, choosing the appropriate styling for the type of standard, and so on. This allows technical committees to focus on specifying new technical features, without having to concern themselves with the mundane requirements of the standards publication process. Further, it also incorporates utilities like syntax highlighting, WebIDL error reporting, and other features that make technical specifications easier to read, use, test, and implement in browsers, such as Firefox, Chrome, Edge, and Safari.

The Web App Manifest specification serves as a typical example of ReSpec-based specification:

ReSpec has been in active development for over 8 years, and is used extensively by Mozillians and the Web Standards Community. There are currently over 70 W3C specifications relying on it, along with an increasing list of standards bodies making use of the project to publish free and open technical standards.

Planning and Proposal

Before you get the funding, MOSS requires you to submit an application. The application asks questions like the description of the project, reason for funding request, and the project plan.

My proposal was to develop new features, fix long standing bugs and improve the documentation, by working full-time for over 3 months.

This was the first time I made a real project plan:

With a little more professionalism, I created a Gantt Chart, which probably looks nicer:

Although the pictures above were not required in the application, they helped me better understand the project and its timeline.

Spoiler: Things often don’t go as planned. Certainly not with a pandemic around.

When you submit the application, you need someone in Mozilla to champion it to the executive council.

A few weeks after submitting the application, I received the approval email! Be patient if you’re applying for the grant: the process is very rigorous, but worth the wait! The program committee was very helpful throughout the application process.

Once the project starts, you also need to submit reports about the project's status from time to time.

Work, Work, Work

The project officially started on May 4, 2020, though I had already begun to work as per the previous schedule of March 23. I’ve been contributing to ReSpec in my spare time for over 2 years, but with this grant, I became a full-time maintainer.

Working on ReSpec means working on multiple repositories - the core tool, its backend APIs and CI solutions. Fortunately, all of them are written in JavaScript so it needs less context switching.

During the grant period, I developed several new features in ReSpec. A notable one is the generation of an index of terms that a specification defines and references from other specifications.

For a large proportion of the grant period, I worked on documentation improvements. The revamped documentation is now searchable and more structured. It includes updated examples and guides, which I hope will add significant value to all users.

If you’re interested in a summary of the work done from the perspective of ReSpec users, read this email in W3C mailing list. There is also a list of activities undertaken to achieve project milestones.

Conclusion

With the MOSS award, gave me the opportunity to develop the proposed ideas and much more. I set out to let ReSpec empower specification authors by reducing the tooling oriented effort it takes while writing a specification. With the improved syntax support, new configuration options, and more helpful error messages, the community is more empowered to collaborate and standardize the next generation of web technologies. And I hope the comprehensive documentation, full of examples and guides, will be invaluable to all ReSpec users.

If you’re interested in contributing to an open source project, I would like to welcome you to ReSpec. We’ve some good first issues, as well as more involving issues, and we would be happy to mentor you!

w3c / respec

A tool for creating technical documents and web standards

Acknowledgements

Thanks a million times to Marcos, for supporting me through the MOSS process, mentoring, code reviews, reviewing this post and being a friend. This project would not have been possible without you.

I would like to express my gratitude to Mozilla and the MOSS program committee, for providing funds to help make open source sustainable. Shout out to Kagami for their insightful code reviews.

Thank you, for reading this far. Stay safe!

A Weird Hack using TypeScript

Sid Vishnoi — Fri, 19 Jul 2019 12:22:38 +0000

Hi! In a project in my last company, I ran into a weird issue with the modern JavaScript build system and to work around it, wrote an even weirder hack.

Sorry if you found the title being click-baity. Alternative title: Using constant assertions in TypeScript to keep constants in sync.

The Problem

The project was created using create-react-app (CRA) with TypeScript and it also had an Express server, that too in TypeScript. The project structure was simple (and an even more simplified version is shown here).

./
  src/
    App.tsx
  server/
    app.ts

then there was a constant.

const options = ["1 day", "1 week", "2 weeks", "1 month", "1 year"];

The constant options, was required in client side to show a list of allowed options in a <select> and the server needed it for validating the requests coming from client. Now using the DRY principle, I shouldn't be writing that same value of options in two files - as the values at two places may get out of sync. So, it needed to be shared between src and server - that's where the problem started.

Turns out, create-react-app doesn't allow imports outside src directory (See this Stack Overflow post), so I can't import options from server directory.

Hey Sid, why don't you do import from src instead?

// server/app.ts
import { options } from '../src/App.tsx';

The build process in server was simple. Take a .ts file, and run it through tsc to create a .js file, keeping the same directory structure.

So, when I ran tsc on my server/app.ts, instead of creating a directory structure like:

./server-build/
  app.js

it ended up creating:

./server-build/
  server/
    app.js
  src/
    app.js

Oops. Messing up build directory or build process just to share a constant? Not cool.

The Hack

A prior day, I came across constant assertions, a new TypeScript feature.

Turns out, in create-react-app, we can't import values outside src directory, but we can import types.

// server/app.ts
// a const assertion
const options = ["1 day", "1 week", "2 weeks", "1 month", "1 year"] as const;
// export the type instead of value
export type Options = typeof options;

// src/App.tsx
import { Options } from '../server/app.ts';
const options: Options = ["1 day", "1 week", "2 weeks", "1 month", "1 year"];

How does it work? Lets say, I change "2 weeks" in server to "3 weeks". What happens? Well, the client fails to compile! It fails as it's expecting options to have type Options, but "3 weeks" doesn't exist in Options at index 2. It works similarly if I change only in src.

So, we've two variables in sync using TypeScript. Not very DRY, but it works™️.

See for yourself: