DEV Community: kyohei

I built Form Builder like Typeform where nobody can see answers in clear text

kyohei — Fri, 12 Aug 2022 04:33:32 +0000

Hey fellow developers!

I've been recently running Hyde, a privacy-tech company, as a co-founder and CTO of it for a little shy of a year.

I feel that OAuth's conception of "scope" should be richer. Right now, there's nothing that prevents OAuth clients from breaking their promises they made in Authorization screens. Not just in practice but in terms of RFC 6749, scopes simply convey a set of kinds of resources authorized, not how they could or couldn't be used.

I don't think this is a problem of OAuth as it's based on the assumption that, in the first place, you can trust OAuth clients in a degree at which they wouldn't break the promises they themselves made. Like, as long as OAuth clients are somewhat famous enterprises, it would critically harm them if they break the promises they made against their customers.

At the same time, it's obvious for savvy people like ones in this community, that there's a bunch of malicious actors who would intentionally break those promises, and it's not easy to distinguish those shady agencies from indie hackers who actually do really interesting works.

In such a context, I've came up with an idea where you would be able to audit, in advance, what computation would happen on your data. I believe this could be established on open standards out there, but for this time, to prove the concept myself right now, I've built PoC tool for both.

So I made a Form Builder like Typeform where nobody can see answers in clear text.

Kyohei

@tnzk

I made a Form Builder like Typeform where nobody can see answers in clear text!

It works like typical form builders on the surface, but unlike, you wouldn't read responses even if you're the owner. You'll see these instead (what is the use of it? will cover later in thread :)

11:18 AM - 09 Aug 2022

It works like typical form builders on the surface, but unlike them, you wouldn't be able read responses from participants even if you're the owner.

I propose two ways to work with those data kept secret.

First one is UI with visualization but as only vector embeddings, not raw data. Vector embeddings keep characteristic in terms of the operations on elements in the set. You would get the idea roughly by referring several word2vec explanations out there.

In nutshell, vector embedding preserve the relations between elements in a set while it compresses (in other words, anonymize in this context) by mapping them onto another set. For instance, if you get an idea of adding like King + Female = Queen, then you would construct an embedding in which (king.x, king.y, king.z) + (female.x, female.y, female.z) = (queen.x, queen.y, queen.z). This allows you to execute certain class of computation being unaware of actual words i.e. sensitive information in this context. Caveats are that this also requires you, the application programmer here, to understand what the embeddings are and how you would operate on them.

Second one is something perhaps a little overloaded in terms of architecture, but simpler to work with. To access the data, you need to make a Docker image. The image will be executed on our end, with JSON files of responses mounted there. Here, you would be able to process the responses in clear text. And it would be supposed to write results to a specific file, from which we would read and provide you via API. Since they (hopefully including you!) can code on them as usual, unlike on the embeddings discussed above.

Kyohei

@tnzk

To access the data, you need to have a Docker image. The image will be executed on our end, with JSON files of responses mounted there. It would be supposed to write results to a specific file, from which we would read and provide you via API.

11:18 AM - 09 Aug 2022

Tutorial and API reference is available here

Any feedback would be warmly appreciated :)

As a next story, I envisage to integrate this on top of OAuth or GNAP; Either by adding a scope/access.actions with specific semantics or returning results in place of tokens. Since GNAP has async mechanism in nature it would fit better.

Integrating Svelte Kit and Prisma without wrapper class to workaround

kyohei — Thu, 21 Apr 2022 10:14:22 +0000

Svelte Kit and Prisma is a go-to stack for me these days. I like its great ergonomics in frontend and less cost of cognitive load from context switches between server sides.

Users of Svelte Kit, Nuxt or other Vite based frameworks haven't been able to import PrismaClient or other exports in the way the documentation suggests¹.

which leads you to see:


import Prisma, { PrismaClient } from "@prisma/client";
                 ^^^^^^^^^^^^
SyntaxError: Named export 'PrismaClient' not found. The requested module '@prisma/client' is a CommonJS module, which may not support all module.exports as named exports.
CommonJS modules can always be imported via the default export, for example using:

import pkg from '@prisma/client';
const { PrismaClient } = pkg;

While there's several workarounds known in the community, they still have some shortcomings like inability to import Enum values or Prisma object for SQL templates nicely.

For the context, when you use Prisma, what you use is not something in @prisma/client, but a code locally generated prisma generate command, which in turn @prisma/client would import inside. Since Prisma need to provide you a custom made API that reflects your schema, it wouldn't able to come as like an ordinary package.

As a result, @prisma/client contains just a line of code or two, to import the custom-made client supposedly generated at the time of deployment. It's really small piece of code:

const prisma = require('.prisma/client/index')

module.exports = prisma

Since this glues the application code and generated code, I would call this as Prisma's glue code.

This works fine for CJS code to require. Similarly, the code that would be transpiled by TypeScript or other bundlers can work with this, since those tools take care at compile time.

The problem is ES modules, who cannot recognize any exports other than default exports.

Svelte Kit (as far as I know also Nuxt or other Vite based frameworks, depending on configs) emits ES modules as their build artifacts, unlike Next that emits CJS bundles, and encounters this problem.

But why?

When an ES module tries to import named exports from other CJS modules, Node recognizes only the exports that explicitly defiend in specific syntax. They would be detected before they get executed with static analysis ²

For better compatibility with existing usage in the JS ecosystem, Node.js in addition attempts to determine the CommonJS named exports of every imported CommonJS module to provide them as separate ES module exports using a static analysis process.

The detection of named exports is based on common syntax patterns but does not always correctly detect named exports.

Because of this, Prisma's glue code effectively reexports just default export from perspective of Node, while TS compiler somehow recognized the named exports. This causes a weird situation where they would work fine in dev server, but fails with the error message like this once they have been bundled for production, as ES modules with static import s.

Fortunately, Node doesn't detect only export.name = value. It seems to support more involved syntax, like ³:

module.exports = {
  ...require('.prisma/client/index'),
}

Which I sent as a pull request to Prisma. I'm not sure when or even if it would be accepted, I assume you would be able to use this hook in package.json for the time being. Assuming above glue code replacement (JS snippet) saved as patches/prisma-fixed-glue.js, add this:

  "build": "cp patches/prisma-fixed-glue.js node_modules/@prisma/client/index.js && svelte-kit build",

(I use pnpm which no longer supports lifecycle scripts in default, but you can put prebuild as long as you enabled it or use npm)

Jest-gauge, a Jest extension to write acceptance tests in natural languages like Gauge

kyohei — Mon, 09 Aug 2021 03:14:57 +0000

TL;DR

I’ve released a Jest extension that allows you to write acceptance tests in natural languages similar to Gauge, in which you can practice acceptance tests driven development; ATDD.

https://github.com/tnzk/jest-gauge

What is Acceptance-Test Driven Development (ATDD)?

Acceptance-Test Driven Development is a software development methodology derived from Test-Driven Development, as known as TDD. In contrast to TDD which focuses on describing what you expect a class or a module to be, ATDD encourages you to describe acceptance criteria of the subject system as a whole, in a larger sense.

Roughly speaking, they provide a feedback cycle of software tests around unit testing with TDD and E2E testing for ATDD, respectively.

Behavior-Driven Development, BDD, is another methodology which also derives from TDD. BDD shares the same feedback cycle as in TDD, but focuses on defining “behaviors” of a class or a module. More importantly, BDD involves a wider variety of stakeholders to participate in. Unlike TDD which is designed and evolved as a practice for programmers to get their job done nicely, BDD puts emphasis on describing User Stories to define what kind of value should be delivered to customers, through “Discovery workshop” by the “three amigos,” said the product owner, developers and testers.

User Stories by theirselves cannot drive a feedback cycle of software tests since they’re just some plain sentences in a natural language. However, expressing an expected behavior directly as a piece of test code will prevent non-programers to participate in the session as stakeholders, which exactly BDD encourages in the first place.

To overcome this dilemma, Cucumber, a well known BDD framework, provides a DSL called Gharkin, which you can describe the insights found in “Discovery workshop”. Gharkin allows you to write something like “AS a Customer, TO avoid standing in the line for a bank teller, I WANT to withdraw cash via ATM”, similar to well-known User Story Template. This looks understandable to stakeholders who don’t code.

As Daniel North explained was inspired by Ubiquitous Language by Eric Evans, they are almost identical to the domain definition, in other words, the "acceptance criteria" to the product from the business owner's perspective. The motivation towards ATDD is the question of how we can drives software development by defining these acceptance criteria in terms of executable specification of the software.

Why Gauge?

Through the explanation above, you might find BDD and ATDD don't seem to differ that much. If so, can't we use Cucumber to drive ATDD, as long as we let the minor mismatches aside?

The dealbreaker is the fact that Gherkin is not a natural language on a closer look, but a loosely-defined formal language. Finished examples in Gherkin looks as if written in a natural language indeed, but when it comes to writing, stakeholders who don't code found it very tough to write in it (Programmers, of course including me, sometimes misbelieve even non-programmers can read and write a simple piece of RSpec or something, but in reality they can't. This should be a cognitive bias which many programmers share I think).

Gherkin's syntax in which you write the behavioral definition is, in fact, not a natural language but slightly formally constrained.

In Gauge, you can write literally in a natural language. While specification file itself is defined as subset of Markdown, sentences inside are just a line of a human language.

# Top page Specification 

You can explain freely the background or motivation of the specification, since paragraphs here will be ignored as just comments.

## Scenario: a user can open a site and see the top page.

- Open "https://duckduckgo.com/".
- It shows a picture of a cute Cucumber-looking bird to the user.

To run tests, Gauge will look up step implementations by the title for corresponding to specification lines, in exact matching basis. You can insert a template variable to give it to some robustness as you like, whereas in turn it makes the sentences look like in a formal language. Unlike Gherkin, you can tweak the strictness of the language by using or not using these variables or other similar mechanisms.

I'm interested in Gauge since it allows us to write specifications and acceptance criteria in a free-form natural language.

Why Jest-gauge?

Gauge seemed a little opinionated to me, and felt it required a heavy-lifting to integrate into an existing product. It was hesitant for me to give a surprise like "Hey guys, we need to switch the testing framework to Gauge!" to team members who just had been familiarized with unit/E2E testing in Jest.

I suppose I will need Gauge in the future to navigate full-fledged ATDD, but my primary motivation at this time is something narrower than "Is ATDD good or not?", more like "Will specs in a Gauge-like natural language be the catalyst that leads communication between developers and non-developer stakeholders?"

So, I've started building a tool for a team with existing unit/E2E tests in Jest to try ATDD easily.

https://github.com/tnzk/jest-gauge

Refer README for installation and usage.

Since it is a Jest extension, you can try it out by adding a configuration into jest.config.js. For specification above, it shows a test report like this:

$ npx jest --config=jest.config.gauge.js specs/

PASS  examples/welcome.spec

Top page Specification 

Scenario: a user can open a site and see the top page.
✓ Open "https://duckduckgo.com/"
✓ It shows a picture of a cute Cucumber-looking bird to the user.

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        0.913 s

It supports a large part of Gauge's functionality, but of course there's many things not supported yet. You can find what is supported and what is not in TODO section in README.

Motivation

I'm running a coding bootcamp in Japan. We are building and maintaining a learning management system for internal use from scratch. We have stakeholders of the project working as a non-tech staffs and they have very raw and factual needs about what matters for operating the educational activity in the bootcamp, as massive and complex Excel spreadsheets! It had motivated me to wander how to keep them organized and ship as working software.

I tried convincing the stakeholders "We're a coding bootcamp, so we should take a risk and try new things on software development methodology. It must lead us to some insights about quality of the educational service". I knew this is a kind of big ask to be accepted in other domain of businesses. Even developers are surprised on this. So I must say a warm appreciation for accepting this to colleagues.

A bit tangentially, that was not the only stretching. We've built a video chat functionality from scratch with WebRTC into the LMS, where the lectures and technical support sessions take place. Screenshot below are us doing a workshop on it. It takes much of efforts to stabilize the connectivity since we don't rely on a famous product like Agora and Zoom APIs, and in course we had learnt many insights and grown team unity.

So, I decided to build Jest-gauge, after concluding Gauge itself is not for our team at this moment and catching up ATDD in 2021 and Cucumber/Gherkin which I experienced half decade ago. And then on, our CI runs acceptance tests written in our mother tongue everyday.

Yes, this is our proud production case :)

A joke aside, we accept risks because we are not a big-co and of the background above. There's many rough edge in it, so please use it at your own risk.

Conclusion

These experience above, and the books on ATDD like "Growing Object-Oriented Software, Guided by Tests", leads me to a belief that it might be too early for developers and other stakeholders to communicate through behavioral descriptions or in a ubiquitous language, in evidence-based manner, a decade ago.

Increasing importance of technology and things like no-code platforms may change this. We may be able to discuss and share what a product is and should be through a semi-formal language in like Gauge or Cucumber.

Members of a company who runs a coding bootcamp, maybe inevitably, looks like having relatively good understanding of computers and software, even they're not developers. This might be the reason we had adopted ATDD with relatively little friction. We'll practice the ATDD with Jest-gauge there at this moment, and later transfer that insight to other fields with less association with software.

But this is just a try. Maybe it works, or maybe not. If you try Jest-gauge, and got some insights, please share it with me. Of course, any contributions are warmly welcomed.

Svelte for Web Components development: Pitfalls and workarounds

kyohei — Mon, 05 Jul 2021 14:09:40 +0000

Svelte components can be compiled to custom elements, aka web components.

Since Svelte is a library in relatively early stage, there are some pitfalls to avoid with workarounds, which I'm going to describe in this article.

Corresponding code for repro and trying out yourself the workarounds are on GitHub. The working example is available online via Vercel.

Attributes named in kebab-case won't be recognized

Every props defined in Svelte components compiles to an attribute of a custom element. In HTML, most of the attributes are named in kebab-case, specified as words in lower alphabets combined with -¹.

In Svelte, however, props are described as a set of declaration of variables, which in JavaScript cannot include - in the name. This is known issues² with a workaround.

Svelte team recognizes this but has not been resolved. It is suggested to use $$props to access the props like $$props['kebab-attr'] in these situations².

This, however, works only in the case you use the custom element in HTML directly. It is okay for the end users of the custom element since they would use it in that way but is problematic for developers of the components. If you mount it as Svelte component, all props should be undefined at that moment the component has been instantiated, unintentionally.

// App.svelte
<script>
import './Kebab.svelte'

let name = value
</script>

<input bind:value>
<swc-kebab your-name={name}></swc-kebab>

// Kebab.svelte
<svelte:options tag="swc-kebab" />

<script>
export let yourName = $$props['your-name']
</script>

Hello, {yourName}

Another workaround which allows you to code <swc-kebab your-name={name}></swc-kebab> is to have a wrapper class to intercept default behavior of the Svelte³:

// KebabFixed.js
import Kebab from  './Kebab.svelte'

class KebabFixed extends Kebab {
  static get observedAttributes() {
    return (super.observedAttributes || []).map(attr => attr.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase());
  }

  attributeChangedCallback(attrName, oldValue, newValue) {
    attrName = attrName.replace(/-([a-z])/g, (_, up) => up.toUpperCase());
    super.attributeChangedCallback(attrName, oldValue, newValue);
  }
}

customElements.define('swc-kebab-fixed', KebabFixed);
// App.svelte
<script>
import './KebabFixed.svelte'

let name = value
</script>

<input bind:value>
<swc-kebab-fixed your-name={name}></swc-kebab-fixed>

Attributes with upper-case letters won't be recognized

Similarly, you cannot use an upper-case letter in the name of attributes if the component is mounted as a custom element. For instance, even you specified like yourName="some value", it will be converted to a lower-case version like yourname.

It seems the browsers that convert names to comply the naming convention explained above, rather than a problem of Svelte's Web Components support.

Since camelCase is de-facto standard way of naming in JavaScript, naming a prop like yourName as usual would result undefined.

In this case, changing two occurrence of yourName to yourname fixes it to work properly. Unlikely, the attribute name on caller side doesn't matter, whichever it is yourName="camelCase" or yourname="non camel case".

// App.svelte
<script>
import './NoUppercase.svelte'

let name = value
</script>

<input bind:value>
<swc-no-uppercase yourName={name}></swc-no-uppercase>

// NoUppercase.svelte
<svelte:options tag="swc-no-uppercase" />

<script>
export let yourName // Change this to `yourname`
</script>

Hello, {yourName} <!-- Change this to `yourname` -->

Changing one of props via DOM API applies to the component, but bind mechanism doesn't work

In the example above, I have used Svelte notations to set attribute values. You can leverage the most of Svelte functionality to develop custom elements. Changes of value propagates to name in the child component which depends to value.

Svelte notation does not available in HTML, so you wouldn't be able to yourname={name}. The only way to set attribute values is to code yourname="a string literal" directly. Use DOM APIs to change these attribute values dynamically:

const element = document.querySelector('swc-child')
element.yourName = 'a updated name'

Whenever attribute values changed, attributeChangedCallback which Svelte registered propagates the change to the internal DOM of the custom element. This enables you to treat the custom element similarly to Svelte components.

On the other hand, there's no support of bind: mechanism in custom elements. Changes in child custom elements will not be available to parent components.

Use custom events I'd described later to pass back the changes in child custom elements. In this case, end users of the custom element must register an event listener to subscribe the events.

This weighs to the end users, but it is reasonable for them to be responsible of since they've decided not to use any front-end frameworks.

You can't pass an object other than a string through attributes

Svelte components accept any objects as contents of props. But attribute values in HTML accept just a literal string.

If you have a Svelte component first and try to compile it to a custom element, this might be a problem. You can serialize an object to JSON if the object is simple enough, while it is very unlikely in the real world.

A (weird) workaround would be to have an object like "store" in global namespace, pass any objects you want through the store. As long as the key is just a string, you can set it to the attribute values of the custom element.

// App.svelte

<svelte:options tag="swc-root" />

<script>
  import PassAnObjectFixed from './PassAnObjectFixed.svelte'

  let name = 'default name'

  window.__myData = {
    'somekey': {}
  }
  $: window.__myData['somekey'].name = name
  const syncToParent = () => {
    name = window.__myData['somekey'].name
  }
</script>

<input bind:value={name}>
{name}
<p>As WC: <swc-pass-object name={data}></swc-pass-object></p>
<p>As Svelte: <PassAnObject {data} /></p>
<p>As WC: <swc-pass-object-fixed key="somekey"></swc-pass-object-fixed><button on:click={syncToParent}>Sync to input field</button></p>

// PassAnObjectFixed.svelte

<svelte:options tag="swc-pass-object-fixed" />

<script>
export let key
let name

const refresh = () => {
  name = window.__myData['somekey'].name
}
refresh()

$: window.__myData['somekey'].name = name

</script>

Hello, {name} <button on:click={refresh}>Refresh</button>
<input bind:value={name}>

This way, the parent component can read the changes the child applied to store, thus you can have some feedback mechanism like the bind: in anyway.

Of course it is not very cool since only the key would be specified explicitly. I'd prefer to change the values through DOM API and custom events to have dependency of data clear.

Emiting a custom event in Svelte doesn't emit a DOM event automatically

Svelte supports custom events to emit any component specific events other than built-in events like on:click, on:keydown or on:focus.

However, a callback set via addEventListener wouldn't be able to catch them since they're built on Svelte-specific event mechanism. In the example below, you can see how a custom event, which is successfully listened in Svelte event handler, doesn't fire the callback registered via addEventListener.

// App.svelte
<svelte:options tag="swc-root" />
<svelte:window on:load={() => handleLoad()} />

import CustomEventExample from './CustomEventExample.svelte'

let name = 'default name'

const handleCustomEvent = (event) => name = event.detail.name

let rootElement
const handleLoad = () => {
  const customElement = rootElement.querySelector('swc-custom-events')
  customElement.addEventListener('namechanged', handleCustomEvent)
}
$: if (customEventElement) customEventElement.name = name
</script>

<div bind:this={rootElement}>
  <h1>Custom Event</h1>
  <p>As Svelte: <CustomEventExample {name} on:namechanged={handleCustomEvent} /></p>
  <p>As WC: <swc-custom-events name={name}></swc-custom-events></p>
</div>

// CustomEventExample.svelte
<svelte:options tag="swc-custom-events" />

<script>
import { createEventDispatcher } from 'svelte';
const dispatch = createEventDispatcher();

export let name
  $: (name) && dispatch('namechanged', { name })
</script>

Hello, {name}
<input bind:value={name}>

A workaround suggested in GitHub³ would be like below. There, you can have a wrapper to emit a DOM event also:

<svelte:options tag="swc-custom-events-fixed" />

<script>
  import { createEventDispatcher } from 'svelte';
  import { get_current_component } from 'svelte/internal';

  const component = get_current_component();
  const originalDispatch = createEventDispatcher();

  const dispatch = (name, detail) => {
    originalDispatch(name, detail);
    component?.dispatchEvent(new CustomEvent(name, { detail }));
  }

  export let name
  $: (name) && dispatch('namechanged', { name })
</script>

Hello, {name}
<input bind:value={name}>

Styles defined in child components doesn't apply

You can use a component as a Svelte component or a custom element almost interchangeably. One of subtle difference would be how a set of styles defined in components applies.

A component with <svelte:options tag="tag-name" /> will have a shadow root.

On the other hand, child components in the above said component won't have a shadow root. The <style> section will be extracted and merged into the parent's one. Thus,

// App.svelte
<svelte:options tag="swc-root" />

<script>
import StylesEncupsulated from './StylesEncupsulated.svelte'
let name = 'default name'
</script>

<h1>Styles</h1>
<p>As Svelte: <StylesEncupsulated {name} /></p>
<p>As WC: <swc-styles-encapsulated name={name}></swc-styles-encapsulated></p>

// StylesEncupsulated.svelte
<svelte:options tag="swc-styles-encapsulated" />

<script>
export let name
</script>

<span>Hello, {name}</span>

<style>
  span { color: blue }
</style>

A simple workaround for this is to use inline style. Svelte compiler does not touch the inline styles, so it keeps existing and applies.

// StylesEncupsulated.svelte
<svelte:options tag="swc-styles-encapsulated" />

<script>
export let name
</script>

<span style="color: blue;">Hello, {name}</span>

But this is not cool since you must code the same styles repeatedly, as well as have scattered template code.

Uncaught (in promise) TypeError: Illegal constructor at new SvelteElement

Svelte use the component classes directly to createElements.define to register custom elements. If you enabled customElement in compiler options, there's no way to control which component should be compiled to a custom element and which is not.

So you'll encounter Uncaught (in promise) TypeError: Illegal constructor at new SvelteElement if you misses <svelte:options tag="swc-styles-encapsulated" /> in any component inside the project.⁴

Install Puppeteer on MacBook Pro with Apple Silicon / M1

kyohei — Sun, 13 Jun 2021 12:47:06 +0000

For those who wouldn't like to work outside the CLI.

Install Chromium via brew:

$ brew install chromium
$ `which chromium`

Set an environemntal variable to prevent [npm/yarn/pnpm] install downloading Chromium automatically (I'd put .env into the repo so coworkers with Apple Silicon can $ source .env .)

export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
export PUPPETEER_EXECUTABLE_PATH=`which chromium`

Although in some situations you need to patch Puppeteer since it has a hard-coded executable path to /usr/bin/chromium-browser, you can try an option of puppeteer.launch to overwrite it.

const browser = await puppeteer.launch({
    executablePath: process.env[‘PUPPETEER_EXECUTABLE_PATH’],
})

References

TDD, BDD, ATDD and Agile timeline

kyohei — Wed, 05 May 2021 07:48:55 +0000

Timeline

2000 Test Driven Development: By Example by Kent Beck
2001 Agile Manifesto
2002 Scrum Alliance
2003 Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans, BDD and JBehave by Daniel North
2003 FitNesse
2004 BDD getting traction among the community
2005 RSpec by Dave, Aslak
2006 Introducing BDD by Daniel North
2008 Cucumber by Aslak
2010 Growing Object-Oriented Software, Guided by Tests
2010 The RSpec Book
2011 The training wheels came off by Aslak
2012 Page Object pattern
2013 Screenplay pattern
2014 TDD is dead. Long live testing. by DHH
2018 gauge by ThoughtWorks

Background and Motivation

I'm not so crucial about BDD, nor a ritual practitioner of it. Even so I occasionally come up with an idea that BDD must be suited to the issues I encountered sometimes.

This time I wish to be a little more thoughtful. I've collected the important events in BDD and Agile field, and sorted them chronologically so I can grasp a brief history and trend of TDD, BDD and ATDD.

Any information and advice would be appreciated, especially events since 2014 which seem missing in this list.

Sources

Trying openapi2aspida to auto-generate Type Definition for a Web API

kyohei — Sun, 31 May 2020 07:49:00 +0000

Previously, we had explored how Aspida provides a way to integrate external Web APIs into your code in type-safe manner.

But writing down the complete type definition of an API by hand hardly seems very fun. Rather, we'd like it to be generated automatically.

openapi2aspida tools gives this functionality, as long as the API provides OpenAPI 3.0 or Swagger schema.

Generate Type Definition

Since you've followed instruction in previous post, you would have apis directory, which this time openapi2aspida expects it to be absent. So we need remove this before trying this tool.

$ rm -rf $REPO_ROOT/apis

Create aspida.config.js, Get JSON schema path

Run this to generate and pour packages into apis/:

$ npx openapi2aspida --build

Try using

$ yarn run aspida-mock --build
> Property 'resHeaders' is missing in type '{ status: 200; resBody: { about: { description: string; title: string; locale: string; version: string; https: true; moderators: { id: number; username: string; avatar_template: string; name: string; title: string; }[]; admins: { ...; }[]; }; }; }' but required in type 'BaseResponse<{ about?: { description?: string; title?: string; locale?: string; version?: string; https?: boolean; moderators?: { id?: number; username?: string; avatar_template?: string; name?: string; title?: string; }[]; admins?: { ...; }[]; }; }, unknown, 200>'

Wow I thought I had tried to compile C++ abusing class templates here.

This error seems to occur because Discourse API doc (perhaps including OpenAPI schema). Discourse maintainers said so in the doc and the thread.

Let the linter explain before deep diving this ourselves:

$ yarn global add ibm-openapi-validator
$ curl -O https://docs.discourse.org/openapi.json
$ lint-openapi openapi.json | head -n 10

[Warning] No .validaterc file found. The validator will run in default mode.
To configure the validator, create a .validaterc file.

errors

  Message :   Parameters MUST have their data described by either `schema` or `content`.
  Path    :   paths./c/{id}.json.get.parameters.0
  Line    :   296

$ lint-openapi openapi.json | wc -l
1318

Conclusion

We have tried Discourse API and found it lacked an OpenAPI-compiliant Schema. While openapi2aspida doesn't seem to work very well in this situation, but if you are in a project where you have well-written OpenAPI schema, openapi2aspida will be promising for your development.

Unfortunately, you wouldn't find so many projects with OpenAPI Schema out there at the moment. Yet we saw previously, we can start writing type definition by hand as necessary.

If you'll happen to be design a web API, you can contribute the OpenAPI ecosystem by starting it with writing OpenAPI Schema for the API. This might be encouraged from test-driven point of view.

Making Web APIs Type-safe with Aspida

kyohei — Thu, 14 May 2020 07:29:06 +0000

Types matter. Even with the naive type system of C/C++, I used to feel somehow protected.

TypeScript enables for JavaScript libraries to provide their type definitions but Web APIs. Actually, majority of APIs lack precise documentation, let alone type definition.

Aspida fills this blind spot. Few HTTP client libraries on NPM provide this capability.

In this article, I demonstrate how we can invoke Discourse API as an example in type-safe manner.

Setting up Aspida

You can setup Aspida according the official README.

Since Aspida provides just an abstraction layer for type-safety, you need to chose one of HTTP client library as its back-end. Major libraries (axios, ky, fetch and node-fetch) seem to be supported.

Here, I pick axios.

Overview of Discourse API

You may know that Discourse is an open source discussion platform.

I try accessing an instance of Discourse via its Web API and show the names of visible Categories in the forum for demonstration.

Discourse API is a simple RESTful API, with nice and sufficient documentation. I'm not so fluent about the API, but it seems to cover almost all of its functionalities.

Since I'm the Server Admin of a Mastodon server, I chose Mastodon Meta Discussion Board for an example :)

Creating Type definition

First of all, we need the type definition.

You can assign types for the response and request parameters of your favorite APIs with Aspida, by putting type definition files in $REPO_ROOT/apis/ like:

export type CategoryList = {
    can_create_category: boolean,
    can_create_topic: boolean,
    draft: boolean,
    draft_key: string,
    draft_sequence: number,
    categories: Category[]
}

export type Category = {
    id: number,
    name: string,
    color: string,
    text_color: string
    slug: string,
    topic_count: number,
    post_count: number,
    position: number,
    description: string,
    description_text: string,
    topic_url: string,
    logo_url: string,
    background_url: string,
    read_restricted: boolean,
    permission: number,
    notification_level: string,
    can_edit: boolean,
    topic_template: string,
    has_children: boolean,
    topics_day: number,
    topics_week: number,
    topics_month: number,
    topics_year: number,
    topics_all_time: number,
    description_excerpt: string,
}

export type Methods = {
    get: {
        resBody: {
            category_list: CategoryList
        },
    }
}

This time I put this as categories.ts.

This is a hand-crafted type definition ™️ looking up the API documentation 💪

Building the Type definition

Once you have created the type definition, you need to build before using in application:

$ yarn run aspida --build

You may be happier by having this defined in package.json.

Invoking the API in Application

Now you can invoke the API in type-safe manner! You can write your application like below.

import dotenv from "dotenv"
import axios from "axios"
import aspida from "@aspida/axios"
import api from "../apis/$api"
import type { CategoryList } from "../apis/categories"

dotenv.config()

const axiosConfig = {
    timeout: 3000,
    baseURL: 'https://discourse.joinmastodon.org',
    headers: {
        'Accept': 'application/json',
//        'Api-Username': process.env.DISCOURSE_API_USERNAME,
//        'Api-Key': process.env.DISCOURSE_API_KEY,
    }
}

let client = api(aspida(axios, axiosConfig))

;(async () => {
    client.categories.get()
        .then(response => {
            let category_list = response.data.category_list
            category_list.categories.forEach(cat => console.log(cat.name))
        })
        .catch(error => console.log(error))
})()

It seems you need to import types explicitly if you want enable code-completion.

Also, you can pass Axios options at instantiation of Aspida. I had confirmed it worked well with headers for authentication.

This results:

$ yarn run start
Server administration
General
Core development
App development
Translation
Meta feedback
Feedback
Done in 10.56s.

Looks good 👍

You can see complete code at: https://github.com/tnzk/aspida-demo

You're now embraced in type system, so you won't break a thing like:

category_list.categories.forEach(cat => console.log(cat.name * 1))

because this will be detected at compile time:

src/discourse-list-category.ts:25:65 - error TS2362: The left-hand side of an arithmetic operation must be of type 'any', 'number', 'bigint' or an enum type.

25             category_list.categories.forEach(cat => console.log(cat.name * 1))

Upnext

Aspida has another interesting functionality which automatically build type definitions from OpenAPI Specification.

Since Discourse API provides it, we'll try this in next article :)