DEV Community: koki kitamura

Server can handle 10 million users

koki kitamura — Sun, 01 May 2022 14:10:26 +0000

Overview

I created an API server that is highly scalable and can handle 10 million users. It's an SNS like Twitter.
The implementation is published on Github.

The development environment is as follows.

Node 16.14
Express 4.17.3
DynamoDB 2012-08-10

The functional requirements are as follows.

Post a tweet
Post a Comment for tweet
Follow user
Get Timeline

Introduction

Services with hundreds of millions of users, such as Facebook, Amazon, and Youtube, need to handle a lot of traffic. A commonly used approach to handling heavy traffic is scale-out rather than scale-up. Scale-up is expensive because it uses high-performance server. In addition, there is a performance limit for operating on one server.

Let's talk about scale-out. Application can be broadly divided into three layers.

Client layer
Server layer
Database layer

When handling a large amount of traffic, The server layer only processes the data, it does not store it. Therefore, it is easy to scale out.

On the other hand, the database layer becomes difficult to maintain consistency and availability as data is distributed due to scale-out. You also need the logic to decide which data is stored on which node. Data relocation is required when increasing or decreasing the number of nodes. Since these features are not in RDB, we will use NoSQL.

Typical databases that support scale-out include BigTable, HBase, DynamoDB, Cassandra, etc.

Database	Description
BigTable、HBase	Consistent and up-to-date data can be obtained. On the other hand, data cannot be acquired while the lock is applied due to data update.
DynamoDB、Cassandra	Data is always accessible. On the other hand, old data may be read during data synchronization.

This time, we will create an API server for SNS, so availability is more important than consistency. Therefore, we use DynamoDB.

What is DynamoDB?

DynamoDB is a key-value database. You can create tables, and each table stores an item. Each item has a key and a value.

You can specify a partition key and a sort key for the item key. The partition key is used to determine the node from within the DynamoDB cluster. The sort key is like an index on a table and is used for sorting.

You can store multiple attribute / value pairs for an item's value. The attributes can be different for each item.

DynamoDB queries are limited and basically narrow down items by partition key and sort key only. When querying using other attributes, it will be slower as the number of items increases because it is necessary to check all items.

When you want to treat other attributes as partition keys, use GSI (Global Secondaly Index). When other attributes are treated as sort keys, LSI (Local Secndary Index) is used.

Database Design

DynamoDB's database design is different from RDB. The flexibility of querying RDBs allows you to design a normalized table first, without considering access patterns to your data. On the other hand, DynamoDB has a limited query pattern, so first determine the access pattern to the data and then design the table based on it. Specifically, we will proceed with the following flow.

Modeling
Create use case list
Design Table
Create query definition

Modeling

The ER diagram is as follows.

The timeline shows tweets of users that you are following. In SNS, the display speed of the timeline has a great influence on usability. Consider a database design that can display the timeline faster.

Read Heavy / Write Light on the timeline

In the case of a normalized table design, writing data at the time of tweeting is light because data is written only to the Tweets table. On the other hand, reading data on the timeline is heavy. The main flow when reading the timeline is as follows.

Get a list of IDs of users you are following
Get tweets from each user you follow
Merge the retrieved tweets
Sort merged tweets

The SQL for getting the timeline is as follows.

SELECT
  *
FROM 
  tweets
WHERE
  userId IN (
    SELECT followeeId FROM follows WHERE followerId = 'user id'
  )
ORDER BY
  postDate DESC

With this method, the more followers you have, the heavier the load on the timeline will be. It can be said to be a Read Heavy / Write Light method.

Read Light / Write Heavy on the timeline

Consider a Read Light / Write Heavy technique. If you create a Timeline table and want to read the timeline, just query the timeline table. On the other hand, when a user tweeted, make sure to write the tweet to the user's follower's timeline.

The SQL for getting the timeline is as follows.

SELECT
  *
FROM
  timelines
WHERE
  userId = 'user id'
ORDER BY
  tweetPostDate

This time, we will use this Read Light / Write Heavy method.

Create use case list

Create a data use case list based on functional requirements to find out how to access the data.

Entity	UseCase	Screen
Tweet	getTimelineByUserId	Home
User	getUserByUserName	User Detail
Follow	getFolloweesByUserId	User Detail
Follow	getFollowersByUserId	User Detail
Follow	getCountFoloweeByUserId	User Detail
Follow	getcountFollowerByUsreId	User Detail
Tweet	getTweetsByUserId	User Detail
Tweet	getTweetByTweetId	Tweet Detail
Comment	getCommentsByTweetId	Tweet Detail

Design Table

We will design the table and index based on the use case list. DynamoDB has a limited query pattern, but a method called Overloading GSI allows for flexible queries.

Include the ID in the sort key. Make the order of ID and record creation time the same. Then you can sort the posts by date without using LSI.

Create query definition

Finally, write out the query conditions. Based on this, we will implement around the database.

Entity	UseCase	Parameters	Table / Index	Key Condition
Tweet	getTimelineByUserId	{ UserId }	Primary Key	GetItem (ID=UserId AND begins_with(DataType, timeline))
User	getUserByUserName	{Username}	GSI-1	Query (DataValue=Username AND DataType=usserProfile)
Follow	getFolloweesByUserId	{UserId}	Primary key	Query (ID=userId AND begins_with(DataType, followee)
Follow	getFollowersByUserId	{UserId}	Primary Key	Query (ID=userId AND begins_with(DataType, follower)
Follow	getCountFoloweeByUserId	{UserId}	Primary Key	Select COUNT / Query (ID=userId AND begins_with(DataType, followee)
Follow	getcountFollowerByUsreId	{UserId}	Primary Key	Select COUNT / Query (ID=userId AND begins_with(DataType, follower)
Tweet	getTweetsByUserId	{UserId}	Primary Key	Query(ID=userId AND begins_with(DataType, tweet)
Tweet	getTweetByTweetId	{TweetId}	GSI-1	Query(DataValue=tweetId AND begins_with(DataType, tweet)
Comment	getCommentsByTweetId	{TweetId}	GSI-1	Query(DataValue=tweetId AND begins_with(DataType, comment)

Design API Server

Software Design

Design based on Domain Driven Design. The layer and directory names are matched.

Directory Name	DDD Layer	Components
src/domain	Domain Layer	Entity / Value Object / Repository Interface
src/application	Application Layer	Application Service / Serializer
src/infrastructure	Infrastructure Layer	Repository / AWS Config
src/presentation	Presentation Layer	API Server

ID generation method

Make the order of ID and record creation time the same. It can be handled by ID generation using the numbering table, but it lacks scalability. Use Snowflake as a scalable ID generation method.

This method divides the bit string into three parts. The ID is the decimal number of this bit string.

Part	Description
Epoch time	The number of seconds of difference from a particular time.
Sequence	It counts up every time an ID is generated and is cleared every second.
Node number	The number assigned to each node.

Implementing Snowflake in Node.js is as following.

import { config } from "@src/config";
import { dateToUnixTime } from "./time";

const workerIDBits = 10;
const sequenceBits = 12;

// Use snowflake
// See: https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake
export class IdGenerator {
  private workerId: number;
  private lastGenerateAt: number;
  private sequence: number;

  constructor(workerId?: number) {
    this.workerId = config.snowflakeWorkerId;
    this.lastGenerateAt = dateToUnixTime(new Date());
    this.sequence = 0;
  }
  generate(): number {
    const now = dateToUnixTime(new Date());

    if (now == this.lastGenerateAt) {
      this.sequence++;
    } else {
      this.sequence = 0;
    }
    this.lastGenerateAt = now;

    // The bit operators ('<<' and '|' ) can handle numbers within
    // the range of signed 32 bit integer.
    return (
      now * 2 ** (workerIDBits + sequenceBits) +
      this.workerId * 2 ** sequenceBits +
      this.sequence
    );
  }
}

FAQ

Is the user's profile information duplicated?

Yes, it's a duplicate. When the profile is updated, you need to start Lambda with DynamoDB Stream to keep it asynchronous and consistent.

Isn't the tweet of a user with many followers a heavy writing load?

Yes, it's expensive. Only when the number of followers is large, it is necessary to take some measures such as dynamically merging when the timeline is acquired without writing to the follower's timeline.

Do you not cache?

Let's do it. It's not too late to monitor and find bottlenecks before making a decision.

Conclusion

In this article, I explained how to create a highly scalable API server. Just keep in mind that excessive performance optimization can go wrong when there are no performance issues.

The implementation is published on Github, so please take a look.

Data Size Real World

koki kitamura — Sun, 20 Feb 2022 13:45:13 +0000

Applications are currently used by people all over the world. The applications have the following features.

Have millions of users
Store a large amount of data that amounts from petabytes to exabytes(TB-EB)
Require performance from ms to μs
Handle millions of requests per second

The number TB-EB may be difficult to understand because it is a number of digits that is not often used in daily life.

Hardware is always required to run the software. It is necessary to select the hardware that is suitable for the features of the software to be run. Understanding the amount of data will help you choose the memory and storage of your hardware. Hardware selection ranges from on-premises server devices and virtual server instance types to PCs used at home.

Today, the advent of the cloud computing blurs the border between application engineers and infrastructure engineers. Many teams of application engineers build infrastructure using the cloud. Understanding the amount of data makes you choose the right hardware for yourself.

In this article, we'll give you a sense of the amount of data by showing the amount of data of various things.

Data size unit

The unit of data size is a byte. Currently, 1 byte is defined as 8 bits.

Since the data size handles a large number of digits, add a prefix to omit the number of digits. In the International System of Units(SI), the prefix is as follows:

Symbol	Name	Factor	Power	EN
k	kilo	10^3	10^3	thousand
M	mega	10^3k	10^6	million
G	giga	10^3M	10^9	billion
T	tera	10^3G	10^12	trillion
P	peta	10^3T	10^15	quadrillion
E	exa	10^3P	10^18	quintillion

On the other hand, the data size prefix is as follows:

Symbol	Name	Factor
B	byte	8bit
KB	kilo byte	1024B
MB	mega byte	1024KB
GB	giga byte	1024MB
TB	tera byte	1024GB
PB	peta byte	1024TB
EB	exa byte	1024PB

The data is treated as a binary number in computer, so the prefix is every 1024, which is the 8th power of 2, instead of every 1000. There is also a notation that uses KiB instead of KB to distinguish it from the International System of Units, but this article uses KB.

Website data size

According to Page Weight, the data size of a website component is as follows:

Name	Size
Total	1.96MB
HTML	31.4KB
CSS	68.9KB
JavaScript	452KB
Font	119KB
Image	956KB
Video	2.07 MB

The target period for aggregation is from January 2017/1 to 2022/1, and the target is mobile sites.

Note that it is the total size per page, not the size per file. Some people may find the size of JavaScript larger than expected. The reason is that it includes not only own code but also the code of external packages such as frameworks and libraries.

File size

The size varies depending on the contents of the file, so it is for your reference. The size and format were converted from the reference as needed. Also, these data are not a comparison of good and bad file formats. This is because the appropriate file format depends on the features of the file.

Name	Size	cf.
Image - small JPG (size 320 x 320)	21.0KB	https://www.instagram.com/p/CZZyelKpmYf/
Image - small PNG (size 320 x 320)	137KB	https://www.instagram.com/p/CZZyelKpmYf/
Image - small WebP (size 320 x 320)	16KB	https://www.instagram.com/p/CZZyelKpmYf/
Image - large JPG (size 1036 x 1036)	187KB	https://www.instagram.com/p/CVoHltuF7_e/
Image - large PNG (size 1036 x 1036)	1.38MB	https://www.instagram.com/p/CVoHltuF7_e/
Image - large WebP (size 1036 x 1036)	148KB	https://www.instagram.com/p/CVoHltuF7_e/
Audio - music MP3 (playback time 3:01)	5.80MB	https://pixabay.com/music/beats-dont-you-think-lose-16073/
Movie - short MP4 720p (playback time 0:09)	851KB	https://www.youtube.com/shorts/Wm3F8kF9WAE
Movie - short WebM 720p (playback time 0:09)	1.10MB	https://www.youtube.com/shorts/Wm3F8kF9WAE
Movie - short GIF 720p (playback time 0:09)	3.50MB	https://www.youtube.com/shorts/Wm3F8kF9WAE
Document - PDF (4 pages)	150KB	-
Document - DOC (4 pages)	100KB	-
Document - XLSX (1000 rows)	140KB	-
Document - PPT (3 pages)	248KB	-
Application - Firefox 97.0.1 (Mac)	364MB	https://www.google.com/chrome/
Application - Discord 0.0.265 (Mac)	193MB	https://discord.com/
Application - Zoom 5.1.1 (Mac)	52.5MB	https://zoom.us/
Application - Xcode 13.2.1 (Mac)	32.1GB	https://developer.apple.com/xcode/

Hardware capacity

The data size of the hardware memory and storage are following table. If there is no standard, the value is shown as a guide.

Name	Size
Memory - AWS EC2 instance t2.micro	1GB
Memory - AWS EC2 instance T2	0.5GB ~ 32GB
Memory - AWS EC2 instance M5	5GB ~ 384GB
Memory - MacBook Pro 13 inch 2020	8GB ~ 16GB
Memory - MacBook Pro 14 inch 2021	16GB ~ 64GB
Memory - iPhone (1st generation)	128MB
Memory - iPhone (13 Pro max)	6GB
Storage - AWS EBS Provisioned HDD	125GB ~ 16TB
Storage - AWS EBS Provisioned IOPS SSD	4GB ~ 16TB
Storage - AWS RDS SSD	20GB ~ 64TB
Storage - MacBook Pro 13 inch 2020 SSD	256GB ~ 2TB
Storage - MacBook Pro 14 inch 2021 SSD	1TB ~ 8TB
Storage - iPhone (1st generation)	4 ~ 16GB
Storage - iPhone (13 Pro max)	128GB ~ 1TB
Storage - Floppy Disk	720KB ~ 1.44MB
Storage - Compact Disk	650 ~ 700MB
Storage - DVD	4.7GB ~ 8.5GB
Storage - Blu-ray	25GB ~ 128GB
Storage - USB memory	32GB ~ 256GB

Real application data volume

According to Data Never Sleeps, the data created by the actual application is following:

Name	Volume per minute	Volume per day	Volume per year
Twitter tweet	575K tweet/min	828M tweet/day	302G tweet/year
Instagram photo	65K photo/min	93.6M photo/day	34.2G photo/year
Slack message	148K message/min	213M message/day	77.8G message/year

Next, let's look at it in bytes. Assuming to tweet on Twitter has 100 characters and one character is 1 byte, 1tweet is 0.1KB. Instagram 1photo is assumed to be 0.1MB. Assuming to message in Slack has 50 characters and one character is 1 byte, 1message is 0.05KB. Based on these things, it is as follows.

Name	Size per year
Twitter tweet	30.2TB/year
Instagram photo	3.42PB/year
Slack message	3.89TB/year

If you operate the service for one year, more than terabytes of data are accumulated. It is difficult to store this amount of data on a single database server, and you need to scale it out and store it on a distributed database server. We call this a distributed database.

There are two ways to build a distributed database: Master/Slave method and partitioning method. The Master/Slave method is an approach to high traffic, not a large amount of data volume. We should use partitioning for a large amount of data.

RDBs are not designed for partitioning. Therefore, maintaining the partitioned RDB is costly. If you want to build a partitioned distributed database, consider a database designed for partitioning, such as DynamoDB.

E2E Test Automation with Autify

koki kitamura — Wed, 11 Aug 2021 12:28:01 +0000

The need for software testing

Software is used in various places around us, such as smartphones, personal computers, home appliances, and automobiles. Software has become an integral part of our lives, and we take it for granted that it works properly. Therefore, if the software does not work properly, you risk losing the credit, time, and money of the company that developed the software.

The need for automation

Software testing is the act of actually operating the implemented software and detecting defects. It is necessary to operate the software under all conditions in order to detect all defects, but it is difficult to verify all conditions because human resources and time resources are limited in the actual. Against this background, the need for software test automation is increasing.

Software test type

The tests are divided into unit tests, integration tests, and E2E test(system tests).
Unit tests are tests on the proguram components(functions, classes, modules, packages, etc.). Usually the developer tests it. Integration tests are tests on system components (API server, microvis, front). Usually tested by the developer. System testing is performed by operating the system in the same way as an actual user. Developers also check the operation, but it is common for a QA engineer other than the developer to guarantee the complex test design and system quality.

Unit testing and integration testing are automated by many teams, with test code running in CI / CD. On the other hand, although there are tools for automating system tests (Selenium, Puppetter, Cypress, etc.), few teams should have continuous system test automation working well.

Test	Team
Unit Test	Developer
Integration Test	Developer
System Test	QA Enginner

System test test case

Many teams have a QA team separate from the engineer team, and engineers should not design test cases and perform system tests. For engineers, the image of a system test often has only a vague image of manipulating the UI and touching the functions to verify the validity of the system. Each screen decides various input value patterns one by one and carefully verifies whether the operation result is as expected.

Test cases to automate

The role of automated system testing is not to guarantee the quality of new features, but to check the degradation of existing features. Since the purpose of automation is to reduce man-hours, it is necessary to automate test cases that reduce man-hours by automation. Test cases that automate and reduce man-hours must satisfy the following inequalities:

(Manual test cost per time)  x (The number of implementations) > (The cost of starting automation) + (Maintenance costs)

The (left side)-(right side) of this formula is the reduction man-hours. Other than the number of implementations, the difference is small depending on the test case, so it is regarded as a constant. In other words, man-hour reduction depends only on the number of tests performed. Testing new features will be focused before the feature is released, but less often after that. Regression testing is performed regularly at the time of release, which is expected to reduce man-hours. In the case of agile development, the cycle of development → test → release is repeated, so it has a high affinity with system test automation. System testing does not automate everything, at most 30% of all test cases. Manual human testing is not gone.

In addition to regression testing, the following may be considered for automation: Let's automate little by little while considering whether it will lead to reduction of man-hours.

Intensive testing of the core of the product
Wide and shallow test of basic functions
Testing for unstable and buggy features
Tests that could not be carried out due to insufficient man-hours

Tools used to automate system testing

There are many tools for automating system tests, but we use Saas Autify, which makes it easy for non-engineers to automate and maintain test cases.

Autify can create test cases by automatically recording browser operations without the need to write scripts. Chrome Extension is used to record browser operations. Entering the url on the recording screen will open a new window for recording.

You can save the test case by performing the operation you want to test in a new window and pressing the save button at the bottom left.

Autify has four concepts.

Scenario
Step group
Test plan
Test Result

A scenario is the smallest unit of test execution in one test case. One scenario is to log in to the application, post an image, send a message to another user, and so on. The scenario consists of multiple steps. The scenario is divided into each step by actions such as pressing a button and URL transition.

Step groups can share some of the scenarios. However, there is a restriction that it is only the beginning of the scenario.

Test plans allow you to group scenarios and run multiple tests. You can also run the test plan in multiple execution environments. You can also run the test plan on a regular basis.

The test result is the execution result of the scenario or test plan. You can see where it failed.

Works with CI / CD

Suppose the release is in a workflow named Deploy in Github Actions. Make sure that the system test workflow starts when the Deploy workflow ends. I want to see the test results and let slack know if it succeeded or failed, so I need to make sure that Autify's test plan is complete. Use autify-cli as the CLI for that. There is also a notification function of Autify's slack, but I made it notification in Github Actions so that it can be divided for each test plan.

name: e2e-test

on:
  workflow_run:
    workflows:
      - Deploy
    types:
      - completed

jobs:
  autify:
    name: E2E Test
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [12.x]

    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - name: Check out source code
        uses: actions/checkout@v1
      - name: Install Autify CLI
        run: |
          curl -LSfs https://raw.githubusercontent.com/koukikitamura/autify-cli/main/scripts/install.sh | \
            sudo sh -s -- \
              --git koukikitamura/autify-cli \
              --target autify-cli_linux_x86_64 \
              --to /usr/local/bin
      - name: Run Autify
        run: |
          response=$(atf run --project-id=${AUTIFY_PROJECT_ID} --plan-id=${AUTIFY_TEST_PLAN_ID} --spinner=false)
          test_result_id=$(jq '.id' <<< "${response}")
          status=$(jq -r '.status' <<< "${response}")

          echo "TEST_RESULT_ID=${test_result_id}" >> $GITHUB_ENV
          echo "TEST_RESULT_STATUS=${status}" >> $GITHUB_ENV
        env:
          AUTIFY_PERSONAL_ACCESS_TOKEN: ${{ secrets.AUTIFY_PERSONAL_ACCESS_TOKEN }}
          AUTIFY_PROJECT_ID: ${{ secrets.AUTIFY_PROJECT_ID }}
          AUTIFY_TEST_PLAN_ID: 99999
      - name: Notify Slack
        run: |
          .github/workflows/notify_slack.sh "E2E Test has finished. Result is ${TEST_RESULT_STATUS}. \`https://app.autify.com/projects/${AUTIFY_PROJECT_ID}/results/${TEST_RESULT_ID}\`"
        env:
          AUTIFY_PROJECT_ID: ${{ secrets.AUTIFY_PROJECT_ID }}

Cases where system test automation fails

Be aware of the following typical cases where system test automation fails.

Automators don't know about test technology
Automated parts that do not reduce man-hours
Large man-hours for automatic test maintenance
The purpose is to automate

Automated testing is aimed at confirming regression, and manual testing for quality assurance is never gone.