DEV Community: jlei523

Is there an opportunity for a Substack + Dev.to hybrid?

jlei523 — Tue, 11 Aug 2020 03:46:38 +0000

Dev.to lets you reach a large audience instantly. But you don't control your audience and brand. 80% of the benefits go to Dev.to. 20% to you.

Substack lets you control your audience and your brand. But you have to slowly build your own audience. However, 80% of the benefits go to you. 20% to Substack.

Is there an opportunity to build a hybrid?

You get your own website (via a subdomain or your own domain name). Then the product aggregates articles, creates nice category pages, and helps you find audiences.

If a Dev.to or Substack employee is interested in this concept, please reach out! I'm thinking of building this hybrid myself or I can join your company to do it.

I made an app that generates unlimited Avatars

jlei523 — Fri, 08 Nov 2019 16:00:07 +0000

GetAvatars.app

Would love to hear your feedback.

This was made using React.js, Next.js, and hosted on Zeit's serverless deployment.

Artwork credit: https://avataaars.com/

Enjoy!

Performance Review: AWS RDS Postgres vs Digital Ocean Postgres

jlei523 — Wed, 23 Oct 2019 20:41:47 +0000

When Digital Ocean announced the availability of a managed Postgres database service, I wanted to answer this question:

Would my app’s performance improve if I switched my Postgres instance from AWS to Digital Ocean?

I'm working on Medtally.com, a data-driven medical community. Because nearly every page on Medtally requires fresh data, the speed of the database is crucial to providing a fast experience for users.

Example page on Medtally. Every page request pulls data from a Postgres database.

Thanks to the generosity of Digital Ocean's Hatch program, I was able to get some free credits to run a test.

Test setup:

AWS (original): Postgres 11 (2 vCPUs, 4GB RAM, 40GB storage, AWS us-west-1)

Digital Ocean: Postgres 11 (2 vCPUs, 4GB RAM, 38GB storage, SFO2)

App hosting: Zeit Now v1 (SFO, AWS us-west-1)

App stack: Express.js, React.js, Next.js (read my tips on building with Next.js)

How performance was measured:

To measure performance as close to the real-world as possible, I tracked the response time of a common query required to fetch data for our popular health condition pages (example).

I used Node.js' perf_hooks API and logged the results.

Here's the example code I used:

const { performance } = require('perf_hooks'); const t0 = performance.now(); //start timer const data = await fetchDataFromDB(id); const t1 = performance.now(); //end timer const responseTime = t1 - t0 //responsetime in ms

Results after ~2,000+ samples:

Average response time:

AWS: 145ms
Digital Ocean: 233ms

Median response time:

AWS: 63ms
Digital Ocean: 164ms

Digital Ocean's response time was 60% longer on average and 160% by median.

The app felt noticeably slower while I clicked around. The Medtally app can load pages in as little as 150ms. Adding 100ms to that time would mean 66% slower - a significant increase.

I was surprised. I expected the performance to be much closer since both databases had the same number of CPUs, RAM size, and disk size.

Some possible reasons why AWS was faster:

The AWS Postgres instance had an advantage in latency because it was hosted in the same physical location as my app (AWS us-west-1 which is San Francisco). However, Digital Ocean's database was also hosted in San Francisco and so the difference should have been negligible and should not have been a 100ms.

Update: The difference is only 2ms after I ran a few pinging tests between the AWS us-west-1 and Digital Ocean SFO2. We can rule this reason out.

Note: I could have eliminated latency as a variable in the test by basing the conclusion on query execution times in pg_stat_statements instead. I did not have enough time to do this but I may revisit in the future.

AWS Postgres simply uses faster hardware (CPU, RAM, SSD).
Digital Ocean's managed Postgres service is new and the company needs more time to optimize its performance. AWS RDS has been in service for 9 years compared to less than a year for Digital Ocean.

Back to AWS RDS

I will be sticking with AWS RDS until Digital Ocean improves its performance.

Besides the performance issues, I really did enjoy using it. The UI/UX is far superior to AWS and so is the documentation. I have high hopes for it in the future and we could use some strong competition in managed database hosting.

With that said, Medtally.com is back to using AWS RDS.

Interesting in connecting with me? You can find me on Twitter.

6 tips: Using Next.js for your next web app

jlei523 — Thu, 29 Aug 2019 05:40:02 +0000

Next.js is indispensable when you want to build a high performance React app.

It’s being used to power some of my apps such as Medtally, a data-driven medical community and True Home, an automated home valuation tool for properties in Hong Kong.

Along the way, I learned a few tricks and figured out a few “gotchas” of the framework that beginners might find useful. And without further ado, let’s get started:

1. You need to cache the data from getInitialProps or the browser’s back button behavior will break.

getInitialProps is used to fetch data for the page but it also fires when the user presses the back button on the browser. This causes the browser to scroll to the position where you previously left off but without the remote data from getInitialProps needed to render. You can read more about this issue here.

To solve this problem, you need to cache the data on the client side after fetching it. Here’s a quick and simple way to do it:


let cache = {};

class App extends React.Component {
  static async getInitialProps({ req, query, asPath, pathname }) {

    let data;
    //if data is in cache then use the cache
    if (cache[someID]) {
      data = cache[someID]
    } else {
      //if not, then fetch from server
      data = await fetch(`someAPI`);
    }

    return {
     data: data
    }

  render() {
      //check if client, if so store the data in the cache. 
      //If you don't do this check, there will be a separate cache stored on the server since Next.js does server side rendering as well.
      if (process.browser) {
        cache[someID] = this.props.data;
      }

      render (){

       //your components
      }
    }
  }

You can also do this with Redux.

2. Use Next.js’ Link component instead of Router.push so Google can crawl your URLs

Google’s crawler doesn’t see links written like this: <div onClick={handleClickWithRouter}>Go to About Page!</div>

So avoid writing your links with Next’s Router if possible. Instead, use Next’s <Link> component like this:

import Link from 'next/link'

function Home() {
  return (
    <div>
      Click{' '}
      <Link href="/about">
        <a>here</a>
      </Link>{' '}
      to read more
    </div>
  )
}

export default Home

3. Next.js works better with Material UI React than Semantic UI React

If you’re trying to choose between Material UI and Semantic UI React component libraries to go along with your Next.js app, you should choose Material UI.

Semantic UI’s responsive components aren’t built in a way that works well with Next.js’ server-side rendering because they look for the browser’s window object which isn’t available on the server.

If you must use Semantic UI, you can hack it together by following this GitHub ticket.

On the other hand, Material UI’s responsive components only use media queries which means your components should render in the same way on the server and in the client.

If I convinced you, head over to Material UI’s Next.js example to get started.

4. If you use isomorphic-unfetch to do your data fetching, you’ll need to provide the absolute URL

The Next.js creators recommend a library called isomorphic-unfetch for data fetching. It has a much smaller bundle size than axios and works well on both the client and the server.

However, isomorphic-unfetch requires an absolute URL or it will fail. I’m assuming it has something to do with the different environments (client & server) on which your code can be executed. Relative URLs are just not explicit & reliable enough in this case.

You can construct an absolute URL from getInitialProps like this:

class App extends React.Component {
  static async getInitialProps({ req, query, asPath, pathname }) {

    let baseUrl = req ? `${req.protocol}://${req.get("Host")}` : "";

    let data = await fetch(baseUrl + 'relativeURL')


    return {
      data: data

    }

  }

Store your URLs in the database if you want the prettiest of pretty URLs

URLs should be pretty so people want to click on them when they see them on Google.

You generally want to avoid having the database table id in your URL like this: /post/245/i-love-nextjs/.

Ideally, your URLs should be something like this: /post/i-love-nextjs.

But this URL is missing the id 245 needed to fetch the data from the database.

To solve this problem, you’ll need to store the URL in the database like this:

id	title	Content	url
245	I love Next.js	because...	/post/i-love-nextjs

On the server, write code to fetch data using the URL in lieu of the post id when someone requests mywebsite.com/post/i-love-nextjs. Here’s an example using Express.js:

server.get('/post/:slug', async (req, res) => {
    const actualPage = "/post";

    //get the data using '/post/i-love-nextjs' as the id
    let data = await fetchDataWithURL(req.params.slug)

    const postContent = { data: data  };
    app.render(req, res, actualPage, postContent);
  });

I recommend slugify if you need a library to convert text into URLs.
If you have many URLs and you’re using a relational database, you should consider adding an index to the url column so that your lookup query runs faster.

CSS breaks in prod but not development when using Material UI with Next.js

function createPageContext() {
  return {
    theme,
    sheetsManager: new Map(),
    sheetsRegistry: new SheetsRegistry(),


    //add this to fix broken css in prod
    generateClassName: createGenerateClassName({
      productionPrefix: "prod"
    })


  };
}

If this happens to you, try adding this line to your getPageContext.js file:

And that’s it!

Search autocomplete for 2 million records with React & AWS CloudSearch

jlei523 — Thu, 29 Aug 2019 05:25:20 +0000

True Home is a bootstrapped web app that provides a home value estimate for every property in Hong Kong — this means over 2 million homes.

Allowing users to look up their home as efficiently as possible became a challenge.

To make it easy for users to find their home, we built a search autocomplete service using the following stack:

AWS CloudSearch
React
react-autosuggest & autosuggest-highlight modules
Express.js server

A few tidbits on Hong Kong real estate:

The majority of the population lives in tall apartment buildings that can have hundreds of units.
Every building in Hong Kong has a unique name like “The Kennedy on Belchers”.
Hong Kongers don’t generally refer to where they live by an address such as “123 Main Street”. Rather, they use their building name and district like “The Belchers Block A in Causeway Bay”.

People in Hong Kong live like this. The middle building also happens to be my current home!

Our search autocomplete requirements:

Can’t use Google Places API because there is no way to connect the results to our database records.
Can’t use Algolia because it’s simply far too expensive for our bootstrapped app ($700 USD/month for 2 million records).
Users should be able to search by the building’s name.
Users should be able to search for their exact unit by the building name and the unit number.

Before we build, we design!

To design the look and feel of our search functionality, I used Sketch App and drew inspirations from where I once worked, Redfin.

Nailing the design early was important because it helped me figure out what tools I needed to use and what data was required.

Now let’s prep the data for AWS CloudSearch

You can prep your data in JSON, CSV, xml or txt formats. We chose JSON because batch uploading only supports JSON and xml formats.

True Home has two search categories: buildings and units.

Here’s an example of how our JSON file looks like:

[
  {
    "buildingaddress": "8 LEUNG TAK STREET",
    "Name": "EIGHT REGENCY (Tuen Mun)",
    "type": "building"
  },
  {
    "buildingaddress": "8 LEUNG TAK STREET",
    "Name": "31/F FLAT N - NA EIGHT REGENCY (Tuen Mun)",
    "type": "unit"
  }
]

Uploading data to AWS CloudSearch

You can upload data in two ways: via the AWS GUI console or via the terminal through aws command.

The maximum file size you can upload to AWS CloudSearch is only 5mb. This presented a problem for us because we had 2 million records totaling 900mb of data to upload!

To solve this problem, we had to generate 180 JSON files, each slightly under 5mb, and batch uploaded them via the aws command line tool.
Here’s the bash script we used to loop through all 180 JSON files and upload to our endpoint:

for VARIABLE in $(ls *.json); do echo $VARIABLE; aws cloudsearchdomain --endpoint-url {ENDPOINT URL here} upload-documents --content-type application/json --documents $VARIABLE; sleep 1s; done

Testing the search results

One nice thing AWS CloudSearch provides is the ability to test search your data immediately in the console.

Here, we can test our newly uploaded data:

Setting up an Express.js endpoint for AWS CloudSearch

Once you’ve verified that you can search your newly uploaded data, let’s spin up an API on the server to query the data. True Home happens to use Express.js.

The data flow works like this:

User types a search → search query is sent to Express server → Express server gets data from CloudSearch endpoint → Express sends search results back to browser

Wait a minute! Why do we need to go through a server? Why not just query the CloudSearch endpoint directly from the browser?

Unfortunately, CloudSearch doesn’t support CORS which means you either have to go through a server like Express.js or set up some kind of proxy service, both of which will add latency to each query.

Luckily for us, the latency hit isn’t too big because our server and CloudSearch instance are hosted in the same AWS location.

Here’s an example of how to set up the Express API:

server.get("/autocomplete/:searchString", async (req, res) => {

    let cloudSearchEndpoint = 'your endpoint here'

    let data = await axios.get(
      `  ${endpoint}/2013-01-01/search?q=~${
        req.params.searchString
      }&return=_all_fields%2C_score&highlight.label=%7B%22max_phrases%22%3A3%2C%22format%22%3A%22text%22%2C%22pre_tag%22%3A%22*%23*%22%2C%22post_tag%22%3A%22*%25*%22%7D&highlight.unitcode=%7B%22max_phrases%22%3A3%2C%22format%22%3A%22text%22%2C%22pre_tag%22%3A%22*%23*%22%2C%22post_tag%22%3A%22*%25*%22%7D&sort=_score+desc`
    );
    res.send(JSON.stringify(data));
  });

CloudSearch has official Javascript support but I had a hard time getting it to work for some reason. AWS documentation, in general, is lacking.
As a workaround, I simply used the auto-generated endpoint from the testing tool as my Express fetch URL.

Building the React component

True Home’s React search component is built with react-autosuggest. We chose this module because it has excellent documentation and easy to follow examples.

Initially, I was worried about the difficulty of highlighting the words as the user typed but autosuggest-highlight made this a breeze.

All in all, the front-end code took about 4 hours to complete. Most of the time was spent on formatting the data from CloudSearch and the rest was spent on styling the component.

Here’s True Home’s search component in its entirety for reference.

The result

https://thumbs.gfycat.com/WhisperedAfraidAoudad-mobile.mp4

Conclusion: Search autocomplete is surprisingly easy to build with modern tools but I wouldn’t use CloudSearch again

The whole feature took about 32 hours to complete — much faster than I initially thought as I had no prior experience with search.

As a comparison, it took well over a month to build Redfin’s search functionality back in 2014 by a far more experienced engineer. Granted, Redfin’s search had more requirements, more data, and more platforms to support.

The most time-consuming parts were prepping the data for CloudSearch and looking up CloudSearch’s awful and sparse documentation.

Amazon doesn’t seem to be improving CloudSearch anymore. The last major update was all the way back in 2013. I suspect that this is due to Elastisearch surpassing Solr (what CloudSearch is based on) in popularity.

If I had to do this over again, I would choose Elasticsearch over CloudSearch because the former has better documentation and supports CORS.

And that’s it!