DEV Community: Sergey Todyshev

Lets build mini Google Drive clone quickly

Sergey Todyshev — Thu, 05 Aug 2021 17:53:01 +0000

TL;DR you can find code at https://github.com/sergeyt/multidisk

I am going to show you how to relatively quickly build a small Google Drive app or kind of that. I know "Google Drive" may sound a bit provocative, but it is done just to raise some chunk of your attention 😄. Also believe me or not but I've done the app within few mornings (yep I am morning person 😄). Below is some details about what I've done and how.

High level idea

The idea is simple. I'd like to build a unified UI for all my drives. For sake of simplicity I selected only few platforms as a starting point:

Uploadcare - nice and quite new one with simple APIs and have free plan
Dropbox - I assume no need any introduction

Lets call it MultiDisk, it is like https://www.odrive.com/, but built for the Web using quite modern tech stack.

What I've done so far

In terms of features the app includes the following capabilities:

Allow managing multiple drives
Uploading, downloading and deleting files
At this point I've implemented drives for Uploadcare and Dropbox, but I'd like to add support for Google Drive, etc. Especially if the app gains some attention

You can just briefly watch this short video to have more vision about the features mentioned above.

Tooling

To build the app I've chosen the following tools:

TypeScript - I prefer having types from the start of any project nowadays. The reason for this could be discussed separately 😄. Maybe just because I started typed programming first
Uploadcare - new one for me, lets learn something new together
Next.js - no need any introduction as well
Material UI - good React framework with plenty of components. Maybe later I'll refactor the app to use Chakra UI since it looking promising to me too

All tools are well known to me except Uploadcare and Dropbox APIs. Great, it is a learning opportunity then.

How I've done this

My steps were:

Project bootstrapping
Drive abstraction
Uploadcare Drive implementation
Dropbox Drive implementation
Few UI pages

That is it. Lets dive in and cover each step with some additional text.

Project bootstrapping

According to this Next.js doc it is easy peasy as just running npx create-next-app command in your shell and you are literally done 😄

Drive interfaces

It is time for pumping this post with some code 😄. I've come up with the following interfaces to abstract any drive and its objects. These abstractions are needed to unify different storage providers.

export type ItemType = "file" | "folder";

export interface Item {
  type: ItemType;
  id: string;
  name: string;
  path: string;
  driveId: string;
}

export interface File extends Item {
  size: number;
  createdAt: Date;
  url?: string;
  download?: () => Promise<any>;
}

export interface Drive {
  options: any;
  provider: string;

  getItems(folderId?: string): Promise<Item[]>;
  deleteFile(fileId: string): Promise<void>;
  deleteFolder(folderId: string): Promise<void>;
}

Implementation of UploadcareDrive

The API docs of Uploadcare can be found at https://uploadcare.com/api-refs/rest-api/v0.5.0/.

We will just use axios to make HTTP requests and I did not find any typed client for my needs. I happy to do a small NPM package for Uploadcare API client. I know about this one, but currently it is more focussed on uploading and downloading files and it does not cover all features of Uploadcare API. Maybe just need to request it on github 😄

import axios from "axios";
import { Drive, File, Item } from "../types";
import { checkResponseOK } from "./utils";

type Options = {
  type: string;
  id: string;
  name: string;
  publicKey: string;
  secretKey: string;
};

export default class UploadcareDrive implements Drive {
  private _options: Options;

  constructor(options: Options) {
    this._options = options;
  }

  get options(): Options {
    return this._options;
  }

  get provider() {
    return this.options.type;
  }

  get id() {
    return this.options.id;
  }

  get name() {
    return this.options.name;
  }

  axios() {
    return axios.create({
      headers: {
        Accept: "application/vnd.uploadcare-v0.5+json",
        Authorization: `Uploadcare.Simple ${this.options.publicKey}:${this.options.secretKey}`,
      },
    });
  }

  async getItems(folderId?: string): Promise<Item[]> {
    const resp = await this.axios().get("https://api.uploadcare.com/files/");
    checkResponseOK(resp);
    return resp.data.results.map(
      (f) =>
        ({
          type: "file",
          driveId: this.id,
          id: f.uuid,
          name: f.original_filename,
          createdAt: new Date(f.datetime_uploaded),
          url: f.original_file_url,
          size: f.size,
        } as File)
    );
  }

  async deleteFile(fileId: string): Promise<void> {
    const resp = await this.axios().delete(
      `https://api.uploadcare.com/files/${fileId}/`
    );
    checkResponseOK(resp);
  }

  deleteFolder(folderId: string): Promise<void> {
    return Promise.resolve(undefined);
  }
}

Implementation of DropboxDrive

Dropbox has a wonderful docs with even interactive playground called API explorer. I've implemented the version of DropboxDrive that used short-lived token without refreshing token capability. Sorry about that, maybe next time I'll find a time to revise that.

Here a code of current version:

import axios from "axios";
import trimStart from "lodash/trimStart";
import { Drive, Item, File } from "../types";
import { checkResponseOK, downloadBlob, trimPrefix } from "./utils";

type Options = {
  type: string;
  id: string;
  name: string;
  accessToken: string;
};

export default class DropboxDrive implements Drive {
  private _options: Options;

  constructor(options: Options) {
    this._options = options;
  }

  get provider() {
    return "dropbox";
  }
  get id() {
    return this.options.id;
  }
  get name() {
    return this.options.name;
  }
  get options() {
    return this._options;
  }

  async getItems(folderId: string = ""): Promise<Item[]> {
    if (!folderId.startsWith("/")) {
      folderId = "/" + folderId;
    }
    const resp = await this.axios().post(
      "https://api.dropboxapi.com/2/files/list_folder",
      {
        path: folderId === "/" ? "" : folderId,
      }
    );
    checkResponseOK(resp);
    return resp.data.entries.map((entry) => {
      if (entry[".tag"] === "file") {
        return {
          type: "file",
          id: trimPrefix(entry.id, "id:"),
          name: entry.name,
          path: entry.path_display,
          createdAt: new Date(entry.server_modified),
          driveId: this.id,
          size: entry.size,
          download: async () => {
            const resp = await this.axios().post(
              "https://content.dropboxapi.com/2/files/download",
              undefined,
              {
                headers: {
                  "Dropbox-API-Arg": JSON.stringify({ path: entry.id }),
                },
                responseType: "blob",
              }
            );
            downloadBlob(resp.data, entry.name);
          },
        } as File;
      }
      return {
        type: "folder",
        id: trimStart(entry.path_display, "/"),
        name: entry.name,
        path: entry.path_display,
        driveId: this.id,
      } as Item;
    });
  }

  async deleteFile(fileId: string): Promise<void> {
    const resp = this.axios().post(
      "https://api.dropboxapi.com/2/file_requests/delete",
      {
        ids: [fileId],
      }
    );
    checkResponseOK(resp);
  }

  deleteFolder(folderId: string): Promise<void> {
    return Promise.resolve(undefined);
  }

  axios() {
    return axios.create({
      headers: {
        Authorization: `Bearer ${this.options.accessToken}`,
        "Content-Type": "application/json",
      },
    });
  }
}

Application UI

MVP version of the app has only few pages and few dialogs:

Home page with list of drives
Drive detail page with list of files
New drive dialog
Confirmation dialog for delete actions

Home Page

We will use SWR for data fetching recommended by Next.js folks. The home has straightforward implementation. Here a full script of the Home Page:

import isEmpty from "lodash/isEmpty";
import useSWR from "swr";
import { getDrives } from "../core/store";
import Loader from "../components/Loader";
import Placeholder from "../components/Placeholder";
import DriveList from "../components/DriveList";

export default function Home() {
  const { data: drives } = useSWR("/drives", getDrives);
  if (!drives) {
    return <Loader />;
  }

  if (isEmpty(drives)) {
    return (
      <Placeholder>
        You don't any drives, but you can create one clicking on ADD DRIVE
        button
      </Placeholder>
    );
  }

  return <DriveList drives={drives} />;
}

Where DriveList component is coded as follow:

import Link from "next/link";
import ListItem from "@material-ui/core/ListItem";
import ListItemAvatar from "@material-ui/core/ListItemAvatar";
import Avatar from "@material-ui/core/Avatar";
import DriveIcon from "@material-ui/icons/Work";
import ListItemText from "@material-ui/core/ListItemText";
import ListItemSecondaryAction from "@material-ui/core/ListItemSecondaryAction";
import List from "@material-ui/core/List";
import { Drive } from "../types";
import DriveMenu from "./DriveMenu";

export default function DriveList({ drives }: { drives: Drive[] }) {
  const items = drives.map((d, k) => (
    <Link href={`/drive/${d.id}`} key={k}>
      <ListItem style={{ cursor: "pointer" }}>
        <ListItemAvatar>
          <Avatar>
            <DriveIcon />
          </Avatar>
        </ListItemAvatar>
        <ListItemText primary={d.name} secondary={d.driveType} />
        <ListItemSecondaryAction>
          <DriveMenu driveId={d.id} />
        </ListItemSecondaryAction>
      </ListItem>
    </Link>
  ));
  return <List>{items}</List>;
}

Drive View page

Also it is simple as it is:

import { useRouter } from "next/router";
import useSWR from "swr";
import { Box } from "@material-ui/core";
import { Widget } from "@uploadcare/react-widget";
import { getDrive } from "../../core/store";
import Loader from "../../components/Loader";
import ItemList from "../../components/ItemList";

export default function DriveView() {
  const router = useRouter();
  const { id } = router.query;

  const { data, revalidate } = useSWR(`/drive/${id}`, async () => {
    const drive = await getDrive(String(id));
    const items = await drive.getItems();
    return { drive, items };
  });
  if (!data) {
    return <Loader />;
  }

  return (
    <>
      <Box m={2} mb={2}>
        <label>Upload a file:&nbsp;</label>
        <Widget
          publicKey={data.drive.options.publicKey}
          onChange={revalidate}
        />
      </Box>
      <ItemList data={data.items} />
    </>
  );
}

Where ItemList is coded as:

import isEmpty from "lodash/isEmpty";
import List from "@material-ui/core/List";
import { Item, File, Folder } from "../types";
import FileItem from "./FileItem";
import FolderItem from "./FolderItem";
import Placeholder from "./Placeholder";

export default function ItemList({ data }: { data: Item[] }) {
  if (isEmpty(data)) {
    return (
      <Placeholder>
        This drive is empty, but you can fill it out with something :)
      </Placeholder>
    );
  }

  const items = data.map((item, k) => {
    switch (item.type) {
      case "file":
        return <FileItem key={k} item={item as File} />;
      case "folder":
        return <FolderItem key={k} item={item as Folder} />;
      default:
        return null;
    }
  });
  return <List>{items}</List>;
}

As you might noticed uploading file is implemented just using Uploadcare widget - another big time saver which is nice.

How I've done Dialogs?

Do you really wanna read the code for dialogs? It should be boring. And maybe it should be enough for this blog post 😄

Anyway you can be gone to the github repo and have some fun consuming additional bits of information 😄

Next Steps

Next time probably I'll try adding more features in the application like:

Google Drive
AWS S3 Drive
Framework-agnostic File Manager web component. Maybe using Angular elements
Folders
Just improving UX like making wizard to create a drive
Better authorization, maybe making official Dropbox or Google apps
Storing drive connections in Mongo Cloud to persist config across different devices (maybe paid feature 😄)

Enjoy! EOF 😄

Link to original post

LinguaBook - React app for learning Basic English

Sergey Todyshev — Sun, 24 Jan 2021 16:33:39 +0000

I am building a micro app for learning Ogden's Basic English.
It is open source project with code available on github.

Under the hood it is heavily using my Just-in-Time scraper (my another open source project). This scrapper parses HTML pages from multiple sources in the browser and the parsed results are displayed on the page.

Here are some ideas for making the app even better:

infinite feed with all words displayed as instagram-like cards
ignoring known words
wrapping it into mobile app using React Native or KaiOS
multi-language support
adding more word lists for advanced users

Enjoy! EOF 😄

Link to original post

Micro app to render your JSON resume

Sergey Todyshev — Sun, 10 Jan 2021 07:28:06 +0000

Last time I've quickly made a React component to render JSON resume.
Now I've extended the CV page to consume external resumes making the page to be a micro application.
And now you can specify src query parameter to specify your resume.
You can simply store your resume as Github Gist like mine one.
For example src=c9dcfa1b37dec07fb2ee7f36d7278105 is an url to display Thomas Davis's resume.

Next possible steps:

support more sections from JSON resume
make the page to be mobile friendly
make the page to be ready for printing

Enjoy! EOF 😄

Link to original post

React component to render JSON Resume

Sergey Todyshev — Sun, 03 Jan 2021 16:24:54 +0000

In this post I am going to show a React component to render JSON Resume.
As usual enough words, see the code 😄

import React from 'react';
import { Github, Linkedin } from '@icons-pack/react-simple-icons';
import { DateTime } from 'luxon';

const Resume = ({ resume, style }) => {
  return (
    <div
      style={{
        display: 'flex',
        alignItems: 'center',
        justifyContent: 'center',
        ...style,
      }}
    >
      <div style={{ display: 'flex' }}>
        <div style={{ width: 300, marginRight: 80 }}>
          <div style={{ display: 'flex', justifyContent: 'center' }}>
            <img
              src={resume.basics.picture}
              style={{ height: 150, borderRadius: '100%' }}
            />
          </div>
          <div>
            <h3 style={{ marginBottom: '0.5em' }}>SUMMARY</h3>
            <p>{resume.basics.summary}</p>
          </div>
          <div style={{ marginBottom: '0.5em' }}>
            <h3 style={{ marginBottom: '0.5em' }}>CONTACT &amp; PROFILES</h3>
            <div>
              <a href={`mailto:${resume.basics.email}`}>
                {resume.basics.email}
              </a>
            </div>
            <div>{resume.basics.phone}</div>
            <div>
              <a href={resume.basics.website} target="_blank">
                {resume.basics.website}
              </a>
            </div>
            {(resume.basics.profiles || []).map((p, k) => (
              <div key={k}>
                <a
                  href={p.url}
                  target="_blank"
                  style={{ display: 'inline-flex', alignItems: 'center' }}
                >
                  <Icon url={p.url} />
                  {p.username}
                </a>
              </div>
            ))}
          </div>
          {isEmpty(resume.languages) ? null : (
            <div style={{ marginBottom: '0.5em' }}>
              <h3 style={{ marginBottom: '0.5em' }}>LANGUAGES</h3>
              {(resume.languages || []).map((item, k) => (
                <div key={k}>
                  <span>{item.language}</span>
                  <Tag>{item.fluency}</Tag>
                </div>
              ))}
            </div>
          )}
          {isEmpty(resume.interests) ? null : (
            <div>
              <h3 style={{ marginBottom: '0.5em' }}>INTERESTS &amp; HOBBIES</h3>
              {(resume.interests || []).map((item, k) => (
                <div key={k}>
                  <span>{item.name}</span>
                  <div>
                    {(item.keywords || []).map((tag, k) => (
                      <Tag key={k} color="secondary">
                        {tag}
                      </Tag>
                    ))}
                  </div>
                </div>
              ))}
            </div>
          )}
        </div>
        <div>
          <div style={{ marginTop: 40 }}>
            <h1>{resume.basics.name}</h1>
            <h5 style={{ textTransform: 'uppercase', marginTop: -20 }}>
              {resume.basics.label}
            </h5>
          </div>
          <div style={{ marginTop: 70, marginBottom: '1em' }}>
            <h3 style={{ marginBottom: '0.5em' }}>SKILLS</h3>
            {(resume.skills || []).map((item, k) => (
              <div key={k}>
                <div>
                  <strong>{item.name}</strong>
                  <Tag>{item.level}</Tag>
                </div>
                <div>
                  {(item.keywords || []).map((tag, k) => (
                    <Tag key={k} color="secondary">
                      {tag}
                    </Tag>
                  ))}
                </div>
              </div>
            ))}
          </div>
          <Education education={resume.education || []} />
          <Experience items={resume.work || []} />
          <Experience items={resume.volunteer || []} />
        </div>
      </div>
    </div>
  );
};

const Icon = ({ url }) => {
  const size = 20;
  const style = { marginRight: 4 };
  const u = url.toLowerCase();
  if (u.indexOf('github') >= 0) {
    return <Github size={size} color="black" style={style} />;
  }
  if (u.indexOf('linkedin') >= 0) {
    return <Linkedin size={size} color="black" style={style} />;
  }
  return null;
};

const Experience = ({ items }) =>
  isEmpty(items) ? null : (
    <div style={{ marginTop: '0.5em' }}>
      <h3 style={{ marginBottom: '0.5em' }}>WORK EXPERIENCE</h3>
      {items.map((item, k) => (
        <div key={k} style={{ position: 'relative' }}>
          {items.length === 1 ? null : (
            <Timeline isLast={k === items.length - 1} />
          )}
          <h3 style={{ marginBottom: 0 }}>{item.position}</h3>
          <h5
            style={{
              margin: '2px 0',
              display: 'flex',
              justifyContent: 'space-between',
            }}
          >
            <a href={item.website} target="_blank">
              {item.company || item.organization}
            </a>
            <Period startDate={item.startDate} endDate={item.endDate} />
          </h5>
          <p>{item.summary}</p>
        </div>
      ))}
    </div>
  );

const Education = ({ education }) =>
  isEmpty(education) ? null : (
    <div style={{ marginTop: '0.5em' }}>
      <h3 style={{ marginBottom: '0.5em' }}>EDUCATION</h3>
      {education.map((item, k) => (
        <div key={k} style={{ position: 'relative' }}>
          {education.length === 1 ? null : (
            <Timeline isLast={k === education.length - 1} />
          )}
          <h3 style={{ marginBottom: 0 }}>{item.institution}</h3>
          <h5
            style={{
              margin: '2px 0',
              display: 'flex',
              justifyContent: 'space-between',
            }}
          >
            <a href={item.website} target="_blank">
              {item.website}
            </a>
            <Period startDate={item.startDate} endDate={item.endDate} />
          </h5>
          <p>{item.area}</p>
        </div>
      ))}
    </div>
  );

const Timeline = ({ isLast }) => {
  return (
    <>
      <div
        style={{
          position: 'absolute',
          width: 24,
          height: 24,
          border: '1px solid var(--textLink)',
          borderRadius: '100%',
          left: -30,
          top: 3,
          padding: 6,
        }}
      >
        <div
          style={{
            width: 10,
            height: 10,
            backgroundColor: 'var(--textLink)',
            borderRadius: '100%',
          }}
        ></div>
      </div>
      {isLast ? null : (
        <div
          style={{
            position: 'absolute',
            width: 2,
            left: -19,
            top: 26,
            bottom: -31,
            backgroundColor: 'var(--textLink)',
          }}
        ></div>
      )}
    </>
  );
};

const Period = ({ startDate, endDate }) => (
  <span style={{ width: 130 }}>
    <span>
      {DateTime.fromFormat(startDate, 'yyyy-MM-dd').toFormat('MMM yyyy')}
    </span>
    <span>&nbsp;-&nbsp;</span>
    <span>
      {endDate
        ? DateTime.fromFormat(endDate, 'yyyy-MM-dd').toFormat('MMM yyyy')
        : 'now'}
    </span>
  </span>
);

const Tag = ({ color, children }) => {
  return (
    <span
      style={{
        backgroundColor: color === 'secondary' ? '#48BFE3' : 'var(--textLink)',
        color: 'white',
        borderRadius: 4,
        padding: '0px 4px 2px 4px',
        marginLeft: 4,
        boxSizing: 'border-box',
        fontSize: 'smaller',
        fontWeight: 'bold',
      }}
    >
      {children}
    </span>
  );
};

function isEmpty(a) {
  return !a || a.length === 0;
}

The final result can be seen here.

Enjoy! EOF 😄

How to wait for animations to complete in Playwright script

Sergey Todyshev — Tue, 15 Sep 2020 08:30:36 +0000

In this post I am going to show a function to wait for animations to complete in Playwright test script.
As usual enough words, just try to apply the code below 😄

async function waitNoMutations(page, selector) {
  return await page.evaluateHandle(function (selector) {
    var list = document.querySelectorAll(selector);
    var elements = [].slice.call(list);
    var config = { attributes: true, childList: true, subtree: true };
    var mutations = 5; // wait at least five intervals

    var observer = new MutationObserver(function () {
      mutations += 1;
    });
    elements.forEach(function (target) {
      observer.observe(target, config);
    });

    var decrementInterval = setInterval(function () {
      mutations -= 1;
      if (mutations <= 0) {
        clearInterval(decrementInterval);
      }
    }, 5); // this quant might be reduced?

    function complete() {
      return mutations <= 0;
    }

    return new Promise(function (resolve) {
      var count = 0;
      var completeInterval = setInterval(function () {
        if (count >= 1000) { // timeout?
          clearInterval(completeInterval);
          observer.disconnect();
          resolve("timeout");
          return;
        }
        if (complete()) {
          clearInterval(completeInterval);
          observer.disconnect();
          resolve(true);
          return;
        }
        count += 1;
      }, 5);
    });
  }, selector);
}

This works, but not in 100% cases 😄

Enjoy! EOF 😄

Link to original post.

How to add style loaders to webpack

Sergey Todyshev — Wed, 05 Aug 2020 07:55:46 +0000

In this post I am going to show how to configure webpack adding support for SASS and CSS files.
It is slightly inspired by this post,
but implemented as standalone module that you can easily to add your project.

Install dependencies

Step one.

Single line script for NPM:

npm install --save-dev autoprefixer css-loader mini-css-extract-plugin postcss-loader postcss-preset-env sass-loader source-map-loader style-loader

Single line script for Yarn:

yarn add --dev autoprefixer css-loader mini-css-extract-plugin postcss-loader postcss-preset-env sass-loader source-map-loader style-loader

Webpack Configuration

Step two. Copy below code to file and name it as webpack.sass.js;

const path = require("path");
const ExtractPlugin = require("mini-css-extract-plugin");

const NODE_ENV = process.env.NODE_ENV || "development";
const isDevelopment = NODE_ENV === "development";

const postcssLoader = {
  loader: "postcss-loader",
  options: {
    plugins: [require("autoprefixer")({}), require("postcss-preset-env")({})],
  },
};

const sassOptions = {
  outputStyle: "expanded",
  includePaths: [path.resolve(__dirname, "node_modules")],
};

const sassLoader = {
  loader: "sass-loader",
  options: { sassOptions },
};

const rules = [
  {
    test: /\.css$/,
    use: [
      "source-map-loader",
      isDevelopment ? "style-loader" : ExtractPlugin.loader,
      "css-loader",
      postcssLoader,
    ],
  },
  {
    test: /\.scss$/,
    use: [
      "source-map-loader",
      isDevelopment ? "style-loader" : ExtractPlugin.loader,
      "css-loader",
      postcssLoader,
      sassLoader,
    ],
  },
];

module.exports = function withSass(config) {
  if (!config.module) {
    config.module = {};
  }
  if (!config.resolve) {
    config.resolve = {};
  }
  config.resolve.extensions = [...(config.resolve.extensions || []), ".scss"];
  config.module.rules = [...(config.module.rules || []), ...rules];
  config.plugins = [
    ...(config.plugins || []),
    new ExtractPlugin({
      filename: isDevelopment ? "[name].css" : "[name].[hash].css",
      chunkFilename: isDevelopment ? "[id].css" : "[id].[hash].css",
    }),
  ];
  return config;
};

Step three. Modify webpack.config.js as following pattern:

const withSass = require("./webpack.sass.js");

// your base webpack config goes here
const baseConfig = {
    /// ...
};

// extend webpack config
const config = withSass(baseConfig);

module.exports = config;

Enjoy! EOF 😄

Link to original post

createState function using React Context and ImmerJS

Sergey Todyshev — Mon, 03 Aug 2020 09:51:14 +0000

In this post I am going to show a function for global state management in React applications
using React Context and ImmerJS.
It is heavily inspired by this post.

Enough words, please read the final code :)

import React, { useContext, useEffect, useReducer } from "react";
import { useRouter } from "next/router";
import produce from "immer";
import mapValues from "lodash/mapValues";

type Children = { children?: any };
type ProviderFC = React.FC<Children>;

type MutationFn<T> = (state: Partial<T>) => void;
type Mutations<T> = {
  [name: string]: (state: T, ...args: any[]) => void;
};

type Update<T, M extends Mutations<T>> = { setState: MutationFn<T> } & M;
type UseStateResult<T, M extends Mutations<T>> = [T, MutationFn<T>, M];
type UseStateFn<T, M extends Mutations<T>> = () => UseStateResult<T, M>;

export default function createState<T, M extends Mutations<T>>(
  {
    initialState,
    loadState = () => Promise.resolve(initialState),
  }: {
    initialState: T;
    loadState?: () => Promise<T>;
  },
  mutations?: M
): [ProviderFC, UseStateFn<T, M>] {
  const StateContext = React.createContext<T>(initialState);
  const UpdateContext = React.createContext<Update<T, M>>(null as any);

  function Provider({ children }) {
    const router = useRouter();
    const [state, update] = useReducer(produce, initialState);
    const mutate = update as any;

    const setState = (newState: Partial<T>) => {
      mutate((target: any) => {
        Object.assign(target, newState);
      });
    };

    const actions = mapValues(mutations, (fn) => (...params: any[]) => {
      mutate((target: any) => {
        fn(target, ...params);
      });
    });

    const loadAsync = async () => {
      const newState = await loadState();
      mutate((target: any) => {
        Object.assign(target, newState);
      });
    };

    // TODO avoid this effect if loadAsync is not defined
    useEffect(() => {
      loadAsync();
    }, [router.pathname]);

    return (
      <UpdateContext.Provider value={{ setState, ...actions } as any}>
        <StateContext.Provider value={state as any}>
          {children}
        </StateContext.Provider>
      </UpdateContext.Provider>
    );
  }

  function useState(): UseStateResult<T, M> {
    const { setState, ...mutations } = useContext(UpdateContext);
    return [useContext(StateContext), setState, mutations as any];
  }

  return [Provider, useState];
}

The createState function can be used as follow:

// auth state
const initialState = {
  id: "",
  name: "",
  email: "",
  is_owner: false,
  is_admin: false,
  is_authenticated: !!getApiToken(),
};

type State = typeof initialState;

const [UserProvider, useUser] = createState({
  initialState,
  loadState: fetchAuthState,
});

export { UserProvider, useUser };

async function fetchAuthState(): Promise<State> {
  if (!getApiToken()) {
    return initialState;
  }
  try {
    const resp = await api.me();
    return {
      ...initialState,
      ...resp.data,
      is_authenticated: true,
    };
  } catch (err) {
    return initialState;
  }
}

// example login page
function LoginPage() {
   const [_, setUser] = useUser();
   const handleSubmit = async () => {
       const resp = await api.login(form);
       setUser({
            ...omit(resp.data, ['token', "token_expired_at"]),
          is_authenticated: true,
       });
   };
   return <div> Login Form </div>
}

The example is truncated a bit to save your time.

This function allows specifying custom typed mutations as an object with functions accepting current state to be mutated. This can be optionally used to decompose a reducer function with big switch into smaller mutation functions. And you don't need to define actions, just define functions with parameters.

Enjoy! EOF 😄

Link to original post

JAMstack comments

Sergey Todyshev — Fri, 24 Jul 2020 08:32:05 +0000

My site is a Jamstack site. It is built using GatsbyJS powered by ReactJS. In this post I gonna show how you can add a free comments to your site with the following pretty simple steps.

Use Reddit as a service that stores comments.
Implement React component that shows reddit Comments.
Integrate component to the blog post template

React Component

We gonna use NextJS useSWR hook to fetch comments from reddit re-post.
Here a code snippet of this component.

import React from 'react';
import useSWR from 'swr';
import moment from 'moment'; // to render relative time of the comment
import ContentLoader from 'react-content-loader'; // to render nice loader

export default function RedditComments({ reddit }) {
  if (!reddit) {
    return null;
  }
  const { data, error } = useSWR(reddit, fetcher);
  if (error) {
    // TODO render error
    return null;
  }
  const comments = data ? (
    data.map((item, i) => <Comment key={i} data={item.data} level={0} />)
  ) : (
    <Loader />
  );
  return (
    <div className="reddit">
      <a className="heading" href={reddit} target="_blank">
        Comments
      </a>
      {comments}
    </div>
  );
}

The full source code is available here.

Component Integration

I've added a reddit metadata to each post having a reddit re-post.
After that I've extended a blog post template to render comment block.

// ...
import Reddit from '../components/Reddit';

const BlogPostTemplate = ({ pageContext }) => {
  let {
    // ...
    // new metadata
    reddit,
  } = pageContext;

  // ... computing more state like translations

  return (
      <Layout>
        {/* main content is omitted */}
        <footer>
            <Reddit reddit={reddit}/>
            {/* other footer blocks */}
        </footer>
      </Layout>
  );
};

And voila it amazingly works.

Enjoy! EOF 😄

Link to original post

Efficient parsing of JSON record sets in Rust

Sergey Todyshev — Fri, 17 Jul 2020 05:28:47 +0000

JSON record set is an array of objects. serde_json crate does not provide fast solution to parse it out of the box, but you can do something I came up with. I use StreamDeserializer and custom handling to skip array symbols like '[' or ',':

use crate::Error;
use async_std::io::SeekFrom;
use byteorder::ReadBytesExt;
use serde_json::{Deserializer, Map, Value};
use std::io::{Read, Seek};

type Record = Map<String, Value>;

pub struct JsonRecords<'de, R>
where
    R: Read + Seek,
{
    reader: &'de mut R,
    is_array_start: bool,
}

impl<'de, R> JsonRecords<'de, R>
where
    R: Read + Seek,
{
    #[inline]
    pub fn new(reader: &'de mut R) -> Self {
        Self {
            reader,
            is_array_start: true,
        }
    }

    #[inline]
    fn skip_array_start(&mut self) -> Result<(), Error> {
        self.skip_char(b'[')?;
        self.is_array_start = false;
        Ok(())
    }

    fn skip_char(&mut self, val: u8) -> Result<(), Error> {
        self.skip_ws()?;
        let c = self.reader.read_u8()?;
        if c != val {
            return Err(Error::Parse(format!(
                "expect '{}', but was '{}'",
                val, c
            )));
        }
        Ok(())
    }

    fn skip_ws(&mut self) -> Result<(), Error> {
        loop {
            let c = self.peek()?;
            match c {
                b' ' | b'\n' | b'\r' => {
                    self.discard();
                    continue;
                }
                _ => break,
            }
        }
        Ok(())
    }

    #[inline]
    fn peek(&mut self) -> Result<u8, Error> {
        let result = self.reader.read_u8()?;
        self.reader.seek(SeekFrom::Current(-1))?;
        Ok(result)
    }

    #[inline]
    fn discard(&mut self) {
        self.reader.read_u8().unwrap();
    }

    #[inline]
    fn read_record(&mut self) -> Option<Result<Record, Error>> {
        let de = Deserializer::from_reader(&mut self.reader);
        let mut stream = de.into_iter();
        match stream.next() {
            None => None,
            Some(t) => match t {
                Ok(v) => Some(Ok(v)),
                Err(e) => Some(Err(e.into())),
            },
        }
    }
}

impl<'de, R> Iterator for JsonRecords<'de, R>
where
    R: Read + Seek,
{
    type Item = Result<Record, Error>;

    fn next(&mut self) -> Option<Self::Item> {
        if self.is_array_start {
            match self.skip_array_start() {
                Err(e) => return Some(Err(e)),
                Ok(()) => {}
            }
        }
        match self.skip_ws() {
            Err(e) => return Some(Err(e)),
            Ok(()) => {}
        }
        loop {
            match self.peek() {
                Err(e) => return Some(Err(e)),
                Ok(c) => match c {
                    b']' => {
                        self.discard();
                        let _ = self.skip_ws();
                        return None;
                    }
                    b',' => {
                        self.discard();
                        self.skip_ws().unwrap();
                        continue;
                    }
                    b'{' => return self.read_record(),
                    _ => {
                        return Some(Err(Error::Parse(format!(
                            "expected '{{' or ']', but got '{}'",
                            c as char
                        ))))
                    }
                },
            }
        }
    }
}

#[cfg(test)]
mod tests {
    use crate::json_records::JsonRecords;
    use crate::Error;
    use std::io::Cursor;

    #[test]
    fn test_json_array() -> Result<(), Error> {
        let json = "[{\"a\": 1}, {\"a\": 2}]";
        let mut cursor = Cursor::new(json.as_bytes().to_vec());
        let array = JsonRecords::new(&mut cursor);
        for rec in array {
            println!("{:?}", rec?);
        }
        Ok(())
    }
}

This allows to avoid allocation of intermediate vector for a bit faster processing of large datasets.

Enjoy! EOF 😄

.NET CIL interpreter in F#

Sergey Todyshev — Wed, 15 Jul 2020 15:30:42 +0000

I've prepared compact .NET CIL interpreter called Fint in pure F# for NskDotNet Meetup #6.

It can run C# programs with the following features at this point:

all control flow statements (if, for, switch, etc)
arithmetic operations including bitwise one
boolean algebra operators
string concatenation
static function calls
Console.WriteLine calls

It can be easily extended with other features.

Some code in this project is just ported from PageFX project I wrote about recently.

It would be nice to try porting Fint to JavaScript using Fable and run .NET binary code right in a browser.
Later we can try to port Roslyn C# compiler to WASM with Blazor compiler and build online tutorial that:

compiles small C# program to .NET PE file in browser
runs .NET PE file in browser
shows CIL code for given method
provides debugging experience of CIL instruction set

This could be a great web resource to help learning .NET code at low-level.

If you like Fint repo please stargaze it on github.

Here is a recording of my presentation. Sorry for speaking so boring 😄

Enjoy! EOF 😄

Declarative web scraper

Sergey Todyshev — Wed, 15 Jul 2020 08:42:45 +0000

TL;DR see the code at https://github.com/lingvograph/dataminer/blob/master/Program.cs#L18

This small program takes YAML file that defines how to parse HTML page and outputs tuples of structured data.

Sample YAML file to extract data from Macmillan Dictionary:

- selector: .PRON
  term: transcription
# tags
- selector: .PART-OF-SPEECH
  uniq: true
  term: tag
- selector: .SYNTAX-CODING
  unique: true
  term: tag
- selector: .DEFINITION
  term: definition
- selector: .EXAMPLES
  term: in
- selector: .PHR-XREF
  term: in
- selector: .synonyms .theslink
  exclude: "..."
  term: synonym
- selector: .audio_play_button
  audio:
    - "@data-src-mp3"
    - "@data-src-ogg"

And the core part of parser looks like:

private static async Task Parse(DataSource src, Input input)
{
    var url = src.Url(input);
    var config = Configuration.Default.WithDefaultLoader();
    var context = BrowsingContext.New(config);
    var doc = await context.OpenAsync(url);
    var root = src.Schema.RootNode as YamlSequenceNode;
    if (root == null)
    {
        throw new InvalidOperationException("Expect list of nodes");
    }

    IEnumerable<object> Exec(YamlMappingNode query)
    {
        var selector = query["selector"].ToString();
        var term = query.Get("term");
        var audio = query.Get("audio");
        var exclude = query.Get("exclude")?.ToString();
        var uniq = query.Get("uniq")?.ToString() == "true";
        var set = new HashSet<string>();
        foreach (var elem in doc.QuerySelectorAll(selector))
        {
            if (term != null)
            {
                var name = term.ToString();
                var text = elem.TextContent.Strip();
                if (exclude != null && text.Equals(exclude)) continue;
                if (uniq && set.Contains(text)) continue;
                yield return $"({name},lang={input.Lang},text=\"{text}\")";
                if (uniq) set.Add(text);
                continue;
            }

            if (audio is YamlSequenceNode seq)
            {
                foreach (var cmd in seq.Children.Select(t => t.ToString()))
                {
                    var val = (cmd.StartsWith("@")
                        ? elem.GetAttribute(cmd.Substring(1))
                        : elem.TextContent);
                    val = val.Strip();
                    if (string.IsNullOrEmpty(val))
                    {
                        continue;
                    }

                    yield return $"(audio,url=\"{val}\")";
                }
            }
        }
    }

    foreach (var child in root.Children.OrEmpty())
    {
        var query = child as YamlMappingNode;
        if (query == null)
        {
            Console.Error.WriteLine("skip node {0}", child);
            continue;
        }

        foreach (var result in Exec(query))
        {
            Console.WriteLine(result);
        }
    }
}

Status

It is prototype and pretty much experimental code. It can be easily modified for your needs

Tools

It is done with:

.NET - it needs dotnet runtime for execution
CSharp - yeah it is implemented in C# language
AngularSharp - core thing to parse HTML and traverse the document tree
YamlDotNet - YAML parser

Enjoy! EOF :)