DEV Community: Essential Randomness

A gentle introduction to REST APIs

Essential Randomness — Wed, 06 Oct 2021 18:41:48 +0000

This article was originally written as part of the BobaBoard documentation.

APIs allow applications (clients) to interact with another application (server) without unfettered access to the underlying data, or to the whole set of operations that can be executed on it. In short, an API is a contract defined by the server that establishes a common language that clients can use to communicate with it. Most operations allowed by APIs boil down to fetching, storing or updating data.

While there are no enforced rules about how APIs should be defined, BobaServer's API implements a special set of API guidelines called REST. Among the advantages of adhering to REST principles, is that REST APIs are easier to reason about, define, and scale. You can read about the REST principles followed by BobaServer in the API guidelines page.

How a REST API works

REST APIs rely on the HTTP protocol, the same browsers use to navigate most webpages. In particular, a REST API defines a set of endpoints (URLs) that the client can access to execute operations. A REST API also defines the semantic of the data the client will send to describe the operation parameters, as well as the one of the data it will receive in response.

For requests, REST APIs rely on HTTP methods to execute different operations on the same resource using a single endpoint associated with it. For responses, REST APIs rely on HTTP Status Codes to communicate the result of the operation.

Additional data required as part of the request or response is referred to as the request/response payload. BobaBoard's API returns responses in the JSON format.

REST API example

As an example, let's define the REST API endpoint that associated with contributions on a thread. This endpoint will live at the /threads/:thread_id/contributions/:contribution_id address.

Here are some operations that might be available to the client:

Fetch the data for a contribution in a thread

The client wants to fetch the contribution with id contribution_123 in the thread with id thread_456. To achieve this, the client sends a GET request to the /threads/thread_456/contributions/contribution_123 endpoint.

Possible responses include:

An HTTP Status Code of 200 with a payload that contains the contribution data.
An HTTP Status Code of 404 if the contribution does not exist.
An HTTP Status Code of 401 if the access requires authentication data and none was found in the request.

GET requests are the default type of request the browser sends when you access a web page.

Create a new contribution in a thread

The client wants to create a contribution with id contribution_123 in the thread with id thread_456. To achieve this, the client sends a POST request to the /threads/thread_456/contributions/contribution_123 endpoint. The POST request would also include the contribution data as a payload.

In practice, the contribution id would likely be determined by the server when creating the contribution. In this case, the POST request would be sent to the /threads/thread_456/contributions/ endpoint instead, and the assigned contribution id would be returned as part of the response payload.

Possible responses include:

An HTTP Status Code of 201 if the contribution was successfully created. This might include a payload that contains the finalized contribution data.
An HTTP Status Code of 401 if creating the contribution requires authentication data and none was found in the request.
An HTTP Status Code of 409 if a contribution with the given id already exists.

POST requests are most often used by browsers when submitting forms.

Update a contribution in a thread

The client wants to update a contribution with id contribution_123 in the thread with id thread_456. To achieve this, the client sends a PUT request to the /threads/thread_456/contributions/contribution_123 endpoint. The PUT request would also include the updated contribution data as a payload.

Possible responses include:

An HTTP Status Code of 201 if the contribution was successfully created. This might include a payload that contains the finalized contribution data.
An HTTP Status Code of 403 if the user is not authorized to update the contribution.
An HTTP Status Code of 404 if the contribution does not exist.

Delete a contribution in a thread

The client wants to delete a contribution with id contribution_123 in the thread with id thread_456. To achieve this, the client sends a DELETE request to the /threads/thread_456/contributions/contribution_123 endpoint.

Possible responses include:

An HTTP Status Code of 204 if the contribution was successfully deleted.
An HTTP Status Code of 403 if the user is not authorized to delete the contribution.
An HTTP Status Code of 404 if the contribution does not exist.

If this article was useful, consider retweeting it!

Documenting Express REST APIs with OpenAPI and JSDoc

Essential Randomness — Sun, 18 Jul 2021 21:25:06 +0000

As usual, this article isn't meant as an in-depth guide, but as a documentation of the what, why, and how of certain architectural choices. If you're trying to achieve the same thing and need help, leave a comment!

Updates

7/20/21: Added "documenting models" section.

Goals & Constraints

To document BobaBoard's REST API.
Standardize (and document) both the parameters and the responses of various endpoints.
The documentation should be as close as possible to the source code it describes.
The documentation should be served through a docusaurus instance hosted on a different server.
(Not implemented): Ensuring endpoints conform to the documented API. While we could use express-openapi-validator, it doesn't currently support OpenAPI 3.1 (issue)
- Consideration: at least at first, we'd like to report the discrepancies without failing the requests. I'm unsure whether this is supported by this library.

Final Result

Architecture Flow

Documentation Page

How To

Packages used

SwaggerJSDoc: to turn JSDocs into the final OpenAPI spec (served at /open-api.json).
Redocusaurus: to embed Redoc into Docusaurus. There are other options for documentation, like any OpenAPI/Swagger compatible tool (e.g. SwaggerUI), but Redoc is the nicest feeling one.

Configuration (Express)

OpenAPI Options

These options define the global configuration and settings of your OpenAPI spec. You can find the OpenAPI-specific settings (i.e. the one NOT specific to Redoc) on the OpenAPI website.



const options = {
  definition: {
    openapi: "3.1.0",
    info: {
      title: "BobaBoard's API documentation.",
      version: "0.0.1",
      // Note: indenting the description will cause the markdown not to format correctly.
      description: `
# Intro
Welcome to the BobaBoard's backend API. This is still a WIP.

# Example Section
This is just to test that sections work. It will be written better later.
        `,
      contact: {
        name: "Ms. Boba",
        url: "https://www.bobaboard.com",
        email: "ms.boba@bobaboard.com",
      },
    },
    servers: [
      {
        url: "http://localhost:4200/",
        description: "Development server",
      },
    ],
    // These are used to group endpoints in the sidebar
    tags: [
      {
        name: "/posts/",
        description: "All APIs related to the /posts/ endpoints.",
      },
      {
        name: "/boards/",
        description: "All APIs related to the /boards/ endpoints.",
      },
      {
        name: "todo",
        description: "APIs whose documentation still needs work.",
      },
    ],
    // Special Redoc section to control how tags display in the sidebar.
    "x-tagGroups": [
      {
        name: "general",
        tags: ["/posts/", "/boards/"],
      },
    ],
  },
  // Which paths to parse the API specs from.
  apis: ["./types/open-api/*.yaml", "./server/*/routes.ts"],
};

Documenting Models

OpenAPI specs can contain a Components section to define reusable models. These are not automatically documented at this stage (workaround issue).

To add models documentation, add the following section to your top-level configuration.



const options = {
  // ...
  tags: [
    // ...
    {
      name: "models",
      "x-displayName": "Models",
      // Note: markdown must not contain spaces after new line. 
      description: `
## Contribution
<SchemaDefinition schemaRef="#/components/schemas/Contribution" />
## Tags
<SchemaDefinition schemaRef="#/components/schemas/Tags" />
`,
  ],
  "x-tagGroups": [
    {
      name: "models",
      tags: ["models"],
    },
  ]
}

Add the OpenAPI endpoint

Configure the Express server to surface your spec through an /open-api.json endpoint. Redocusaurus will use it to retrieve the data to display.



import swaggerJsdoc from "swagger-jsdoc";

const specs = swaggerJsdoc(options);
app.get("/open-api.json", (req, res) => {
  res.setHeader("Content-Type", "application/json");
  res.send(specs);
});

Component Specs

Reusable types used throughout the documentation.
/types/open-api/contribution.yaml



# Note the /components/schemas/[component name] hierarchy.
# This is used to refer to these types in the endpoint
# documentation.
components:
  schemas:
    Contribution:
      type: object
      properties:
        post_id:
          type: string
          format: uuid
        parent_thread_id:
          type: string
          format: uuid
        parent_post_id:
          type: string
          format: uuid
        secret_identity:
          $ref: "#/components/schemas/Identity"
      required:
        - post_id
        - parent_thread_id
        - secret_identity

Endpoint Documentation

This should be repeated for every API endpoint you wish to document.



/**
 * @openapi
 * posts/{postId}/contribute:
 *   post:
 *     summary: Replies to a contribution
 *     description: Posts a contribution replying to the one with id {postId}.
 *     tags:
 *       - /posts/
 *       - todo
 *     parameters:
 *       - name: postId
 *         in: path
 *         description: The uuid of the contribution to reply to.
 *         required: true
 *         schema:
 *           type: string
 *           format: uuid
 *     responses:
 *       403:
 *         description: User is not authorized to perform the action.
 *       200:
 *         description: The contribution was successfully created.
 *         content:
 *           application/json:
 *             schema:
 *               type: object
 *               properties:
 *                 contribution:
 *                   $ref: "#/components/schemas/Contribution"
 *                   description: Finalized details of the contributions just posted.
 */
router.post("/:postId/contribute", isLoggedIn, async (req, res) => {
  // The endpoint code
}

Configuration (Docusaurus)

You must update your docusaurus configuration after installing Redocusaurus:
docusaurus.config.js:



module.exports = {
  // other config stuff
  // ...
  presets: [
    // other presets,
    [
      "redocusaurus",
      {
        specs: [
          {
            routePath: "docs/engineering/rest-api/",
            // process.env.API_SPEC is used to serve from localhost during development
            specUrl:
              process.env.API_SPEC ||
              "[prod_server_url]/open-api.json",
          },
        ],
        theme: {
          // See options at https://github.com/Redocly/redoc#redoc-options-object
          redocOptions: {
            expandSingleSchemaField: true,
            expandResponses: "200",
            pathInMiddlePanel: true,
            requiredPropsFirst: true,
            hideHostname: true,
          },
        },
      },
    ],
  ],
}

Sending Subscribers Surveys with Ghost, PipeDream and AirTable

Essential Randomness — Tue, 29 Jun 2021 06:10:12 +0000

Note:

This is not an "how to" guide, but me explaining my own pipeline to myself as a coherence check, and so I can remember what I have running in the future. If you want to deploy a similar project and have questions, you know where the comment section is.

Problem

I've recently created a newsletter with a peculiar gimmick: every month(?) paid users (members) receive a survey to decide the topic of the following month. I want to automate this process as much as possible.

To solve:

Members should receive a new survey each month
When a user becomes a member mid-month, I want to send them any active survey.
Keep track of which members have answered which surveys.

Final Pipeline

Here's an overview of the pipeline. Unfortunately, Excalidraw's sharing features aren't working right now, so I can't link to a larger version.

Technologies

Ghost: My CMS of choice, with integrated membership support. I self-host it on my own server, and it connects directly to my Stripe account.
PipeDream: Makes it easy to create automations spanning a series of different APIs. Similar to Zapier, but you can run custom code if needed. Generous free tier.
SurveySparrow: A survey software. This is the first time I'm using it, moving away from Google Forms. Most important: supports webhooks and variables.
MailerSend: Send email through API. I was already using this for RobinBoob, so adding a new template for the monthly survey was easy.
AirTable: To save the data for future reference. I started with Google Sheets, but soon realized I needed a less-brittle solution to feel comfortable with this pipeline.

Getting Member Data into AirTable

1) When a new user signs up as a member, their data is stored in Ghost's own internal DB. Technically, a webhook event is emitted at this stage, but I couldn't get it to work for my purpose (mostly because I couldn't figure out whether they were a paying member upon subscription).
2) Every two hours, a scheduled PipeDream job uses the Ghost API to fetch members and store them in AirTable. This job is independent from the rest of the pipeline, so I can use this table for all future membership-based actions.

Sending Surveys

1) Every two hours, a scheduled PipeDream job fetches all surveys with status in progress, and all the members who are eligible to receive them. Then, it creates a list of all the non-surveyed members for each survey.
2) For each user, PipeDream uses the MailerSend API to send an email with the survey template. Surveys have a subscriber_email parameters, which is used to pre-fill (and skip) the email step of the survey.
3) For all successful emails sent, the member is then added in the sent_to column of the Surveys table. This prevents repeated emails.

Tracking Answers

1) SurveySparrow triggers a PipeDream webhook when a survey gets a new answer.
2) The webhook triggers the "Register Survey Answer" pipeline, which uses the subscriber_email param to retrieve the member that filled the survey. If the param is not present, the survey will display a question asking for the email explicitly, and that will be used instead.
3) The Member with the corresponding email is added to the answered_by column in the Surveys table.

Potential Improvements

Trigger the Update Members job some time after each webhook event is triggered.
Trigger the Send Survey job whenever an update is made in AirTable rather than (or in addition to) via cron.
While AirTable also has built-in automations, its runs-per-month are too few for my use on the free plan (100). The pipeline could be simplified if I ever switch to the Plus plan, which features 5000 per month.
It's possible for someone to use the subscriber_email param to fake submitting the survey as someone else. This isn't a huge deal for my use case, but a workaround would be to create a unique identifier per survey sent that can then be traced back to the original users.
Rather than make a new survey manually each month, the survey could be automatically generated through the SurveySparrow API.

Adventures in Hacking Electron Apps

Essential Randomness — Fri, 01 Mar 2019 04:15:03 +0000

Disclaimer: Hacking apps is often against the Terms of Service. This article is purely theoretical and not an endorsement of the practice. Always Hack Responsibly.

Have you ever been extremely annoyed by a certain aspect of an app or website?

If you're anything like me, the answer is probably yes. If you're a lot like me, the answer is "yes, often."

On the web, I can easily solve most grievances thanks to extensions that allow you to inject custom CSS and Javascript on any website. With desktop apps, however, I usually have to live with my pain, hoping the developers will one day see the light and decide to fix the problem (or will finally have the time to prioritize doing so).

Unless the app is an Electron app.

I don't remember how I learned about this possibility—sometimes, there's no stronger driving force than the need to fix bad design.

In this article, I'm going to talk about how I changed Discord's app code to solve one of my (and other's) biggest grievances: its huge minimum window size.

Discord's *huge* minimum window size (940px).

Let's be clear, though: this post—just like any worthy software endeavor—is about the journey rather than the solution. So follow along my adventure, see how I figured out the needed steps, and learn something about the command line, electron apps and hacking!

Note: while I use macOS High Sierra, this process can be similarly replicated on other operating systems.

Unraveling Discord's Secrets

Initially, I wanted to learn as much as I could about the Discord application. MacOS has a UI mechanism to explore application contents, which is a good "step zero." Manual exploration only goes so far, though, and soon I turned to the command line.

Step 1: What does Discord's Process Look Like?

To answer my own question, I ran the following command:

ps x | grep Discord

In short, ps lists all the processes running, x includes the ones not attached to a shell (e.g. those started by clicking on application icons), and piping this output (|) to the grep command displays only the ones featuring the string Discord. You can learn more at explainshell.com.

Here's the output, edited for readability:

1927   ??  S      0:00.08 /Applications/Discord.app/Contents/Frameworks/Electron Framework.framework/Resources/crashpad_handler 
    --no-rate-limit --no-upload-gzip 
    --database=/var/folders/sm/4v5p46v175d3x94qp56r37340000gn/T/Discord Crashes 
    --metrics-dir=/var/folders/sm/4v5p46v175d3x94qp56r37340000gn/T/Discord Crashes 
    --url=http://crash.discordapp.com:1127/post 
    --handshake-fd=73

Process #1927

1928   ??  R     34:58.78 /Applications/Discord.app/Contents/Frameworks/Discord Helper.app/Contents/MacOS/Discord Helper 
    --type=renderer --no-sandbox --autoplay-policy=no-user-gesture-required 
    --force-color-profile=srgb --enable-features=SharedArrayBuffer 
    --disable-features=MacV2Sandbox --service-pipe-token=5494336596696404231 
    --lang=en-US 
    --app-path=/Applications/Discord.app/Contents/Resources/app.asar 
    --node-integration=false --webview-tag=false --no-sandbox 
    --preload=/Users/essential_randomness/Library/Application Support/discord/0.0.254/modules/discord_desktop_core/core.asar/app/mainScreenPreload.js 
    --background-color=#2f3136 --num-raster-threads=2 --enable-zero-copy 
    --enable-gpu-memory-buffer-compositor-resources 
    --enable-main-frame-before-activation 
    --service-request-channel-token=5494336596696404231 --renderer-client-id=6

Process #1928

The first process (#1927) seemed to be related to reporting app crashes. I assumed this because of .../crashpad_handler in the app path, and the url flag pointing to http://crash.discordapp.com:1127/post (which is probably the server endpoint crash reports are communicated to).

The second process (#1928) was more promising. In particular I found the value of the app-path variable (/Applications/Discord.app/Contents/Resources/app.asar) worth exploring.

Step 2: Extracting the App Code.

The .asar extension of the file intrigued me. After a quick google search for "asar files", I found a GitHub repo explaining the format:

Asar is a simple extensive archive format, it works like tar that concatenates all files together without compression, while having random access support.

Luckily, the repository contained information on how to install the asar command- line utility (npm install asar) and how to extract archived files from asar files. My next step was both obvious and easy.

Before making any changes, however, I decided to backup the original archive:

# Backup original file in case of emergency
cd /Applications/Discord.app/Contents/Resources/
cp app.asar app_safe_copy.asar

# Extract app.asar to a folder named "unpacked"
asar extract app.asar unpacked/

After cracking this metaphorical treasure chest open, it was time to list (ls) its content!

cd unpacked
ls
# output:
> app_bootstrap common        node_modules  package.json

ls app_bootstrap/
# output:
> Constants.js       bootstrap.js       ...       appSettings.js       ...

The familiar node_modules/ + package.json combination pointed to the archive being an npm package, bundling together a bunch of JavaScript files and other things like images. This was clearly important code! Even better, the code was not compiled or encrypted in any way.

For the first time I thought that (maybe) I could really pull this off!

Step 3: What is This Code?

I explored the JS files by opening them in VsCode to get some insight on the app's structure. This was interesting, but very slow.

To be faster, I decided to bet on a simple assumption: any file controlling the window width would have had to contain the string "width" itself!

Also, I could exclude the node_modules folder from my search because npm packages reserve this directory for external libraries.

# Find all the files containing the string width in the current folder,
# but exclude the ones in the node_modules one.
grep -iRl "width" ./ | grep -v node_modules
# Output:
> .//app_bootstrap/splash/index.js
> .//app_bootstrap/splash/variables.json
> .//app_bootstrap/splashScreen.js

In depth explanation of the command.

This output was disappointing: the files were clearly related to the splash screen, which wasn't what I was looking to change. I tried going up in the top-level Discord folder (/Applications/Discord.app/) and ran the command again, but the output wasn't much different.

It seemed my luck had ran out.

Step 4: Sometimes Stepping Back is a Step Forward

Rather than despair, I decided to go back to process #1928. The preload flag held another interesting path, in a completely different location than the previous one: /Users/essential_randomness/Library/Application Support/discord/0.0.254/modules/discord_desktop_core/core.asar/app/mainScreenPreload.js.

It was time for another adventure!

# Once again, I searched for files containing the string "width".
cd /Users/essential_randomness/Library/Application\ Support/discord/
grep -iRl "width" ./ | grep -v node_modules
# Output
> .//Preferences
> ...
> .//settings.json
> ...
> .//0.0.254/modules/discord_desktop_core/core.asar

This was when I found out you can grep the content of asar files without unpacking them first. Forgetting steps can lead to interesting discoveries.

This search yielded quite a number of files, so I decided to try to narrow it down further. Since I was looking to change the minimum width, I figured any related variable name would be called either minWidth or min_width. After all, we all take code readability seriously, don't we?

grep -iRl "min_width" ./ | grep -v node_modules
# Output
> .//0.0.254/modules/discord_desktop_core/core.asar

grep -iRl "minWidth" ./ | grep -v node_modules
# Output
> .//0.0.254/modules/discord_desktop_core/core.asar
> .//0.0.254/modules/discord_voice/discord_voice.node

core.asar looked really promising! Once again I extracted it and searched for the right file:

cd 0.0.254/modules/discord_desktop_core
cp core.asar core_safe_copy.asar
asar extract core.asar core_unpacked
cd core_unpacked

# Trying min_width first, as the value is likely a constant.
# Constants use, by many code conventions, a capitalized style (i.e. "MIN_WIDTH").
grep -iRl "min_width" ./ | grep -v node_modules
# Output:
> .//app/mainScreen.js

Could .//app/mainScreen.js finally be The One? I immediately opened it, searched for "min_width" and...

const MIN_WIDTH = settings.get('MIN_WIDTH', 940);
const MIN_HEIGHT = settings.get('MIN_HEIGHT', 500);

...JACKPOT!

Step 5: The Moment of Truth

I knew I had to be onto something. With no clue about whether it would work (but tons of faith) I edited the code:


const MIN_WIDTH = settings.get('MIN_WIDTH', 0);
const MIN_HEIGHT = settings.get('MIN_HEIGHT', 0);

Now all I needed to do was repack the changed asar file. Again, I made sure to create a backup of core.asar (cp core.asar core_safe_copy.asar) before going further. Bricking Discord completely was a real possibility here!

With huge trepidation, I ran the final step:

# Remove the original app file and swap it with our edited code, repacked.
rm core.asar
asar pack core_unpacked core.asar

At this point I restarted Discord, hoping the changes would take effect. I placed my cursor at the app border, started dragging and... IT WORKED!

Discord with minimum window size removed

Side Quest: "I Also Like to Live Dangerously"

At this point, I had still one last curiosity. What would have happened if I had messed up while modifying the code?

I re-extracted the asar file, purposely inserted an invalid JavaScript statement, repacked it and tried to run the app. Unsurprisingly, I got an error!

This hammered down the importance of back ups. Since I had wisely created a copy of core.asar, I simply put the original code back in its place and the error was gone.

Conclusion

As a final warning, modifying code this way is likely against most Terms of Service (cue the usual Great Power => Great Responsibility speech).

Another aspect of code hacking to think about is side effects: Discord is not optimized to be displayed at lower sizes and the UI can be wonky. Since Electron apps use Chromium as a fronted, I modified the UI myself through the Developer Tools console (which Discord kindly makes available under "view > developer > developer tools").

Please remember that a very valid reason why developers don't want their code modified is that it can cause unexpected bugs in the app itself. If you choose to run a custom version of any code, don't go filing bugs unless you can reproduce them in the original app!

A Funny Epilogue

After going through all this, I found out there's an easier way to change Discord's window size that does not require modifying the source code.

But, you know, where would the fun be in that?

Happy Hacking!