DEV Community: Soichi Takamura

Stop reinventing server middlewares and take advantage of Express ecosystem

Soichi Takamura — Sat, 21 Mar 2020 16:14:05 +0000

TL;DR

flow-middleware creates a portable server handler from any of Express middlewares. You can run all the adapters of express-session and Passport.js stategies in your Next.js, micro and Node native server today.

import flow from 'flow-middleware';

const handle = flow(
    // Pass Express middlewares
    cookieParser(),
    session({ secret: 'x' }),
    flash(),
    (_reqProxy, resProxy, next) => {
        resProxy.redirect('/');
    }
);

// Handles Node native IncomingMessage and ServerResponse
handle(req, res);

Checkout the Next.js example with Passport.js integration.

Why

As people start using a new Node server library other than Express, they encounter a lack of middlewares that Express already has, which has been well tested and production-ready many years ago. Some of them try to shape a brand new ecosystem on the new island and some just sail back to Express.

Let's start from admitting Express is one of the most successful, beautifully designed and battle-tested software in the Node-based software. Don't forget its hundreds of outstanding middlewares have been born upon it.

Then why you can't use them? The answers will be summarized into these two:

It just can't run since they depend on Express's req.param() and res.redirect(). I don't want to hack to make them work in my ${Your favorite server name comes here}.
Pollution. Express officially recommends middlewares to extend object properties such as req.session = {} and req.flash = fn, where my ${Your favorite server} leaves them clean. Plus, dynamic extensions don't fit today of the TypeScript era.

Yeah, yeah. Then how to solve them both?

How

JavaScript Proxy.

Wrapping req and res by Proxy that splits Node native and Express properties to be accessed. Express exports clean prototypes we can intercept middleware's calls with. It also lets middlewares to call native methods like res.writeHead() and res.end() so native objects properly embed HTTP info like headers and send the response back.

The handler, in the end, returns the proxy objects with like req.session and req.user so you can still use them after the middlewares go through.

Getting started

Install it with Express.

yarn add flow-middleware express

flow(...middlewares)

A function flow creates an http handler from some Express middlewares, processed from left to right of arguments.

import flow from 'flow-middleware';
import { ok } from "assert";
import { createServer } from 'http';
import cookieParser from 'cookie-parser';
import session from 'express-session';
import flash from 'express-flash';

// Creates an async function that handles req and res.
const handle = flow(
    cookieParser(),
    session({ secret: 'x' }),
    flash(),
    (reqProxy, _resProxy, next) => {

        // Our wrapped objects provide accessors
        // that Express middlewares extended💪
        ok(reqProxy.cookies);
        ok(reqProxy.session);
        ok(reqProxy.flash);
        next();
    }
);

createServer(async (req, res) => {

    // Let's run the Express middlewares🚀
    const [ reqProxy, resProxy ] = await handle(req, res);

    // Native objects are clean thanks to our proxy✨
    ok(req.cookies === undefined);
    ok(req.session === undefined);
    ok(req.flash === undefined);

    // You still can access to Express properties here🚚
    ok(reqProxy.cookies);
    ok(reqProxy.session);
    ok(reqProxy.flash);
    ok(resProxy.cookie);
    ok(resProxy.redirect);

    res.end('Hello!');
}).listen(3000);

Check out compose() function to flexibly share middlewares between multiple endpoints.

Conclusion

It takes years for a new framework to have bug-free middlewares for all the sessions, OAuth and logger adapters. Why don't you take advantage of Express ecosystem, one of the most stable solutions for Node.js?

react-apollo-loader: Enhance React, Apollo, TypeScript and GraphQL Utilization

Soichi Takamura — Sun, 08 Dec 2019 15:16:37 +0000

Edit: I solved the problem of tsc failure before webpack build in another package graphql-let. I recommend to use it instead.

I just published an npm package of webpack loader.

👉 react-apollo-loader
👉 react-apollo-loader-example, the example web app of react-apollo-loader

If some of the cases apply to you, you may like it.

You like Apollo
You use Apollo Client with TypeScript
You have a valid GraphQL server
You are willing to have typed GraphQL response

React Hooks to access data by GraphQL Code Generator

GraphQL Code Generator is awesome, especially in combination with React Apollo. When you have such a GraphQL document (= a POST body of GraphQL query),

query News {
  allNews {
    title
    link
    content
  }
}

GraphQL Code Generator elegantly turns it into React Hooks functions like this:

export function useNewsQuery(baseOptions?: ApolloReactHooks.QueryHookOptions<NewsQuery, NewsQueryVariables>) {
  return ApolloReactHooks.useQuery<NewsQuery, NewsQueryVariables>(NewsDocument, baseOptions);
}

After you have it, you can use it inside of your function components.

const Home = (_: Props) => {
  // It's typed⚡️
  const { data, loading } = useNewsQuery();

  return (
    <ul>
      {data?.allNews.map(item => <li>{title}</li>)}
    </ul>
  );
};

To obtain the useful functions, you have to prepare for scanning GraphQL documents and execute graphql-codegen command. You also need to import the generated .tsx, and finally, you can use the data accessing functions.

That is the most flexible and fancy way to bind your GraphQL data and React components today. Here next, what if we can import the functions directly from .graphql?

Loader, a power of webpack 💪

The best way that the current Front-end technology offers to connect the source and the output would look like this.

import { useNewsQuery } from './news.graphql'

To achieve it, let's pull all the requirements together.

We need both GraphQL schema (= the definitions of what the GraphQL server can respond) and GraphQL documents (= the queries of what you want from the GraphQL server).
GraphQL Code Generator needs configurations of how your GraphQL documents should be compiled.
The generated content from .graphql is .tsx, which should be available only in the compiled output. No one wants the physical .tsx files.
Note that .tsx should also be compiled into .js.
We want .d.ts so that the IDEs can guess what functions/types the .graphql will return.

These are exactly what my react-apollo-loader does/allows behind the compilation when webpack detects GraphQL documents import. Check out the Setup section of react-apollo-loader to complete your preparation.

Caveats

There are cases you have to care for. Think about this circular dependency.

To import the Hools function, you need to compile .graphql
To compile .graphql, you need GraphQL schema
To have GraphQL schema, you have to launch GraphQL server
To launch GraphQL server, you have to compile the whole project including .graphql

This happens when you develop an app compiling both GraphQL server and Client-side together, such as a Server-Side Rendering app. In this case, you may want to proceed development step by step, where you first scaffold the GraphQL routes and generate schema.graphql (the static file version of GraphQL schema) and use it to compile import 'myQuery.graphql'.

Another thing you'll encounter is tsc failure. Note that our webpack compilation will generate .d.ts files, which your tsc command requires. So you may want webpack compilation first, and run type checking afterward. Although this is the same anytime you use GraphQL Code Generator, please note and don't waste your time to set up.

Thanks

As always, thanks for reading, I'd be happy if you make feedback👍

pipeline-pipe: Fun Way to Get Your Batch Done with Node Stream

Soichi Takamura — Sun, 27 Oct 2019 07:08:19 +0000

Node Stream is a great way to process iterables. pipeline(), landed in Node v10, makes it flexible and powerful even more.

Here I'd like to share pipeline-pipe, utilities work perfectly with Node Stream, utilities such as pipe() to make your transformer capable to work in parallel.

const { pipeline, Readable } = require('stream');
const pipe = require('pipeline-pipe');

pipeline(
    Readable.from([1, 2, 3]),
    pipe(async postId => await getPost(postId), 16),
    pipe(json => parseHTML(json.postBody).document.title),
    pipe(title => title.includes('important') ? title : null),
    pipe(async title => await storeInDB(title), 4), 4)
    (err) => console.info('All done!')
);

TL;DR

Readable.from + pipeline is a good choice to finish your batch tasks
Use pipeline-pipe for parallel execution
Use Node Stream, not Promise.all

How do you process iterable?

Processing data is our daily job. It often starts from iterable such as an array of RESTful API responses or CSV records to be processed one by one. It then stores the result in a database or somewhere. You may have used for for such tasks.

const sources = [2, 3, 4];

for (let e of sources) {
  const processed = await doSomething(e);
  const result = await storeSomewhere(processed);
  console.log(result);
}

But with Node Stream, it becomes more flexible and efficient.

await pipeline(
    Readable.from(sources),
    pipe(doSomething),    // ← in parallel, with limit
    pipe(storeSomewhere), // ←
    pipe(console.log),
);

I'm going to explain why it's possible step by step in the rest of the article.

Native Node Stream syntax

Let's start by rewriting the above with Node native stream.

const { pipeline, Readable, Transform, Writable} = require('stream');

const sources = [2, 3, 4];

pipeline(
    Readable.from(sources),
    new Transform({
      objectMode: true,
      transform(e, _, callback) {
        doSomething(e).then(processed => callback(undefined, processed));
      },
    }),
    new Writable({
      objectMode: true,
      write(processed, _, callback) {
        storeSomewhere(processed).then(result => {
          console.log(result);
          callback(undefined);
        });
      },
    }),
    (err) => console.log('all done')
);

A bit wordy but it works perfectly. The pipeline(), landed in Node v10.0.0, tells us the end of the stream by callback and sweeps the stream instances by destroying them. The Readable.from(), which was backported to Node v10.17.0, creates a readable stream from iterable such as Array and Map.

In case you're wondering how to get an index of an array in Node Stream, I'll leave this for the tip. Readable.from() is designed great.

// Use index of the array
pipeline(
    Readable.from(sources.entries()),
    new Transform({
      objectMode: true,
      transform([index, e], _, callback) {
        doSomething(e).then(
          processed => callback(undefined, [index, processed])
        );
      },
    }),
    new Writable({
      objectMode: true,
      write([index, processed], _, callback) {
        storeSomewhere(processed).then(result => {
          console.log(result);
          callback(undefined);
        });
      },
    }),
    (err) => console.log('all done')
);

To shorten the execution time

We can improve it. Suppose both the doSomething() and storeSomewhere() take 1 second to execute respectively. The whole process ends up taking 4 seconds since one stream instance can handle one piece of data at one time.

It is solved by parallel-transform by @mafintosh, the author of pipeline and pump. It'll run in parallel during data processing, so the whole process reduces the time to 2 seconds. It's 2 times faster🚀! The code would be below. However, unfortunately, it doesn't work properly now.

const parallel = require('parallel-transform');

pipeline(
    Readable.from([2, 3, 4]),
    parallel(10, (e, callback) => {
      doSomething(e).then(processed => callback(undefined, processed));
    }),
    parallel(10, (processed, callback) => {
      storeSomewhere(processed)
      .then(result => {
        console.log(result);
        callback(undefined);
      })
    }),
    (err) => console.log('all done'),
);

The reason it goes wrong is that it emits the pipeline callback before finishing because of an unsolved bug. So with respect, I published a scoped package @piglovesyou/parallel-transform that fixes it, thanks for @Tappi's PR.

Introduction of pipeline-pipe

Besides the bug fix, there is one more thing I'd like to improve; syntax. By getting rid of calling callbacks and accepting Promise, Node Stream gets a good fit for the asynchronous paradigm.

const { pipeline, Readable } = require('stream');
const pipe = require('pipeline-pipe');

pipeline(
    Readable.from([2, 3, 4]),
    pipe(async (e) => await doSomething(e)),
    pipe(async (processed) => {
      const result = await storeSomewhere(processed);
      console.log(result);
    }),
    (err) => console.log('all done'),
);

It can be even shorter.

pipeline(
    Readable.from([2, 3, 4]),
    pipe(doSomething),
    pipe(storeSomewhere),
    pipe(console.log),
    (err) => console.log('all done'),
);

Plus, feel free to use a promisified version of pipeline(), exported by pipeline-pipe as a utility function.

const {pipe, pipeline} = require('pipeline-pipe');

// ...

  await pipeline(
      Readable.from([2, 3, 4]),
      pipe(doSomething),
      pipe(storeSomewhere),
      pipe(console.log),
  );
  console.log('all done');

Here are gifs to describe how parallel execution brings efficiency to complete 50 tasks, taking 100ms for each. While the sequential for-await example takes 5,000ms+, the parallel pipeline-pipe one only takes 500ms+ (10 times faster).

for-await	pipeline-pipe

Why you shouldn't use Promise.all

Promise.all(sources.map(...)) would be the first option for you to shorten the execution time, but I don't positively recommend it for this reason:

The length of the source is often uncontrollable; it can get 1,000+
Execution processes usually contain asynchronous jobs
When it's RESTful API calls, 1,000+ simultaneous connections could happen
That could be a burden on the API server and could be a factor of making your execution fail

To achieve fast and stable execution, we want a limitation of parallelism. And for Promise friendly syntax, pipeline-pipe would be your option for various kind of batching process execution.

I hope you'll try pipeline-pipe for your next batching project and give me feedback😁

react-apollo + Codegen + TypeScript, How You Can Compose Multiple Queries/Mutations to a Component

Soichi Takamura — Sun, 07 Apr 2019 05:24:33 +0000

Apollo team already provides decent documentation that lets you type your components with types generated from Codegen.

This is a quick tutorial to go further where you compose multiple GraphQL Query/Mutation to bind to a single component.

Summary

Define from outer HOCs (e.g. withData function) to inner HOCs
Pass with-data-props to a child HOC
Do not use compose() function

Example

SUPPOSE this is a new student creation form that requires pulldown schools list. Schools are resolved through GraphQL Query. Also, form submission is done through GraphQL Mutation.

// UserCreate.tsx

import {ChildDataProps, ChildMutateProps} from 'react-apollo';
import {
  CreateUserMutation,
  CreateUserMutationVariables,
} from './__generated__/CreateUserMutation';
import { SchoolsQuery } from './__generated__/SchoolsQuery';
import { CreateUserInput } from "../../../../__generated__/globalTypes";

type ComponentProps = { title: string, };
type WithDataProps = ChildDataProps<ComponentProps, SchoolsQuery, {}>;
const withData = graphql<ComponentProps, SchoolsQuery, {}, WithDataProps>(gql`
  query SchoolsQuery {
    schools {
      id
      email
      nameJa
      nameEn
      phone
    }
  }
`);

type WithMutateProps = ChildMutateProps<WithDataProps, CreateUserMutation, CreateUserMutationVariables>;
const withMutate = graphql<WithDataProps, CreateUserMutation, CreateUserMutationVariables, WithMutateProps>(gql`
  mutation CreateUserMutation($input: CreateUserInput!) {
    createUser(input: $input) {
      email
      nameEn
      nameJa
      school
    }
  }
`);

const UserCreate = withData(withMutate(props => {
  const {
    mutate: createUser,
    data: {schools},
  } = props;
  return (
    <Container>
      <h1>Create New User</h1>
      <Form
        method="post"
        onSubmit={async (e: any) => {
          e.preventDefault();

          const input = (createFormData(e.target) as CreateUserInput ) ;

          const result = await createUser({
            refetchQueries: [
              ({query: UsersQueryGQL} as PureQueryOptions)
            ],
            variables: {input},
          });
          if (!result || !result.data) throw new Error('Mutation failed.');

          const {
            data: {createUser: created},
          } = result;

          // Do something with created user data

          history.push('/users');
        }}
      >
        <FormGroup>
          <Label for="email">User email address:</Label>
          <Input id="email" type="text" name="email"/>
        </FormGroup>

        <FormGroup>
          <Label for="nameJa">User name (Japanese):</Label>
          <Input id="nameJa" type="text" name="nameJa"/>
        </FormGroup>

        <FormGroup>
          <Label for="nameEn">User name (English):</Label>
          <Input id="nameEn" type="text" name="nameEn"/>
        </FormGroup>

        {schools && (
          <>
            <FormGroup>
              <Label for="school">school</Label>
              <p>
                <select id="school" name="school">
                  <option>---please select---</option>
                  {schools.map(s => (
                    <option key={s.id} value={s.id}>
                      {s.nameEn}
                    </option>
                  ))}
                </select>
              </p>
            </FormGroup>
          </>
        )}

        <FormGroup>
          <Button color="primary" type="submit" block>
            Log in
          </Button>
        </FormGroup>
      </Form>
    </Container>
  );
}));

export default UserCreate;

In Outer HOC

type WithDataProps = ChildDataProps<ComponentProps, SchoolsQuery, {}>;
const withData = graphql<ComponentProps, SchoolsQuery, {}, WithDataProps>(/*...*/);

The first argument in generic arguments of graphql() is used as actual acceptable properties
The WithDataProps is used as types we pass to the next inner component
- So this is what we should accept in the inner withMutate.

In Inner HOC

type WithMutateProps = ChildMutateProps<WithDataProps, CreateUserMutation, CreateUserMutationVariables>;
const withMutate = graphql<WithDataProps, CreateUserMutation, CreateUserMutationVariables, WithMutateProps>

Accept props WithDataProps, passed from the outer HOC withData
- which already includes ComponentProps, so the final props the component consumes is compound of 3: Query data types, Mutation types and component own props.

`compose` function?

As of react-apollo version 2.5.2, compose is not typed.

export declare function compose(...funcs: Function[]): (...args: any[]) => any;

I wrote this post because I couldn't find any example to do that. Hope it helps someone.

Learning Apollo through React Starter Kit

Soichi Takamura — Tue, 04 Sep 2018 15:53:12 +0000

Apollo toolset is pretty fascinating these days. It's a relatively big project based on GraphQL spec, that is because the Apollo development team provides many separated modules for user's incremental adoption, instead of so-called a full-stack monolith framework. The benefits are huge.

However, because it's kind of hard to get the whole picture of Apollo and how each module relates, especially when you assume server-side rendering (SSR) with it, we've gotta have a good example project. Here, this one would be your option.

👉 react-starter-kit:feature/apollo-pure

It includes:

apollo-client and react-apollo
apollo-link-state for client-side state management
apollo-server to serve GraphQL API
SSR with fluent data passing from server to client

React Starter Kit (RSK) is a React+Express based SSR boilerplate of web development. It's not only a highly capable template for a general web service but also a concrete educational material of modern frontend development. Especially these lines in server.js are beautiful as I've read a hundred times. It's like a crystal of how web development evolves. Thus I trust RSK and pushed an Apollo example upon it.

There already was an apollo feature branch by @langpavel from which I've learned a lot. The main difference between these is that mine doesn't include Redux code, being replaced by apollo-link-state. Further, the feature/apollo-pure uses apollo-server so that you will grasp an overhead view from frontend to the backend of an Apollo project.

One challenge for me this time is to design a comfortable/easy-to-understand placement of a state role directory. Although Apollo provides a well-considered way to transparently access to both client-side state and remote data through GraphQL syntax, these are different for some reasons. So I separate these in src/data (concerning feature/apollo by @langpavel) and src/state which all client state and its resolvers go to.

As always, thank you for reading this, and I'll appreciate more if you review the branch and make a comment on this PR. I may miss some necessary functionalities of Apollo in it. I'd like to make it a real Apollo boilerplate. Please tell me what it is. Thanks👍

react-kindness: A customizable screen guide for React

Soichi Takamura — Sun, 05 Aug 2018 09:44:02 +0000

I just published a simple screen guide component of React. Please take a look.

👉 react-kindness Demo
👉 react-kindness README

The motivation of the development is thought that using SVG mask and pointer-events: none should be able to implement screen guide simple. Other libraries often place 4 absolutely-positioned dark divs around a focused element, which determines it cannot draw a circle spot in the overlay. SVG mask of course can.

Another implementation makes a target element to be position: relative; z-index: 9999999 !important; to let users "touch" the element and hide any other elements with an overlay div, which seems a cool solution at a first glance. However, I wonder whether we have to block user interactions during a screen guide. Rather, users may want to click an anchor link or click a bell icon to check recent notifications on the page, don't they?

react-kindness is a simple UI guide that doesn't block user interaction. Just shows a spot in an overlay, a message panel containing "prev" and "next" buttons and that's it.

Here is a minimum example of usage.

import {KindnessPanel, Kindness} from 'react-kindness';
import 'react-kindness/dist/index.css';

// ...
    <KindnessPanel
        enabled={this.state.show}
        onExit={() => this.setState({show: false})} />

You probably want to switch on/off the guide, so keep the state within your component as this.state.show above. Please don't forget to disable it on a onExit event hook.

<Kindness>
    <input type="text" {...} />
</Kindness>

<Kindness message="Click here to submit your post!">
    <button type="submit">
        Submit
    </button>
</Kindness>

Then wrap your elements with <Kindness /> which a spot will track. The other attributes like title and order are also available. Please refer to README for the documentation.

About blocking user interactions, if you feel like it, you can return false in onClickOutside. react-kindness will take care of disabling the user action for you.

    <KindnessPanel
        // ...
        onClickOutside={() => false} />

And next, this is another thing I wanted to try, a "function as a component" pattern of React Component development. Using that, the panel content of react-kindness is fully customizable.

Internally, the panel content is implemented as <KindnessPanelContent /> that renders a message and buttons. Instead of using the default content, you can render arbitrary elements inside the panel.

import Button from '@material-ui/core/Button';

// ...

<KindnessPanel enabled={true}>
    {
        ({message, totalSize, currentIndex, goNext, goPrev}) => (
            <div>
                <h3>{message}</h3>
                <h4>{`You are seeing ${currentIndex + 1} of ${totalSize} in the guide.`}</h4>
                <Button action={goPrev}>Go previous</Button>
                <Button action={goNext}>Go next</Button>
            </div>
        )
    }
</KindnessPanel>

Also, the styles of the panel and spot can be modified as you wish.

Finally, I'd like to say I appreciate a project Popper.js which react-kindness heavily depends on. Its functionalities of DOM positioning are comprehensive and sophisticated so that I could read and trace the code easily and utilize the functions that Popper.js exports for external developers like me. The architecture to modify positioning step by step in modifiers is just cool and extensible. If you face problems of DOM positioning, try Popper.js 👍

DEV Community: Soichi Takamura

Stop reinventing server middlewares and take advantage of Express ecosystem

TL;DR

Why

How

Getting started

flow(...middlewares)

Conclusion

react-apollo-loader: Enhance React, Apollo, TypeScript and GraphQL Utilization

React Hooks to access data by GraphQL Code Generator

Loader, a power of webpack 💪

Caveats

Thanks

pipeline-pipe: Fun Way to Get Your Batch Done with Node Stream

TL;DR

How do you process iterable?

Native Node Stream syntax

To shorten the execution time

Introduction of pipeline-pipe

Why you shouldn't use Promise.all

react-apollo + Codegen + TypeScript, How You Can Compose Multiple Queries/Mutations to a Component

Summary

Example

In Outer HOC

In Inner HOC

compose function?

Learning Apollo through React Starter Kit

react-kindness: A customizable screen guide for React

`compose` function?