DEV Community: Mathew Chan

Scheduled Vertical Scaling of AWS Aurora with EventBridge & Lambda in CloudFormation

Mathew Chan — Wed, 28 Feb 2024 08:38:29 +0000

Aurora under heavy write workloads

The following is the database load metrics of our Aurora cluster under peak loads with auto-scaling enabled.

We adjusted the size of our instances after each test to see how much they would throttle under load.

db.t3.medium instances - extreme throttling
db.r6g.large instances - moderate throttling
db.r6g.xlarge instances - no throttling

In short, auto-scaling in Aurora does not handle heavy writes.

Note: even with throttling, the application may still receive reasonable response times from the DB because the t3 family is a burstable instance, but that will incur CPU credits and extra cost.

Problem: handling periodical heavy writes

You can scale out an Amazon Aurora cluster by adding read replicas, or scale it up by using larger instances.

Auto-scaling on instance-based Aurora only supports scaling out, but not scaling up. While extra read replicas help with increased read workloads, it doesn't help with increased write workloads.

Since there is only one writer endpoint (master), the only way to handle heavy write workloads are by replacing the master instance with an instance of a larger size. That being said, updating the master instance size will cause unwanted downtime for our database.

This article will present an approach to using EventBridge to schedule a scale-up of an Aurora writer instance with minimal downtime. Traffic will be simulated with load tests using k6 and verified against RDS performance metrics.

Note on Aurora Multi-Master and Aurora Serverless v2

While Aurora multi-master was once a feature for Aurora MySQL 5.6, it has since been deprecated, and if you are using Aurora for postgreSQL like myself, you're out of luck.

On the other hand, Aurora serverless can scale for increasing write workloads. Serverless v1 wasn't suitable for ongoing web services given its long wakeup time from idle (~30 seconds), and serverless v2, while not having a cold start issue, is prohibitively expensive.

From: AWS Aurora Serverless V2 — What’s new?

Each GB of Serverless V2 RAM is twice the price of V1 and more than 3 times the price of provisioned Aurora capacity (Sam Gibbons)

That is not to mention that

you can reserve instances in regular Aurora for even more cost savings.
RDS proxy is more expensive on serverless v2 unless you have half a dozen of replicas.

As there's always an instance running, v2 is more a managed service than a serverless offering. It all feels like a marketing ploy to align it with pay-per-use services like Lambda.

Other AWS services like DynamoDB or Redis on cluster mode support write scaling by default, so it was quite surprising to me to learn that write scaling wasn't a built-in feature for Aurora.

Scheduled Scale-up for Aurora

Schedule a lambda function to run before write peaks
The function launches a db.r6g.xlarge instance to the Aurora cluster.
On instance creation, an RDS event triggers a Lambda function.
The function fails over the cluster to the newly created instance.

Scale up - Launching a new instance to the Aurora cluster

Lambda function for launching a db.r6g.xlarge instance to the cluster



const AWS = require('aws-sdk');
const rds = new AWS.RDS();

exports.handler = async (event) => {
    try {
        const params = {
            DBClusterIdentifier: process.env.DBClusterIdentifier,
            DBInstanceIdentifier: process.env.DBInstanceIdentifier, // rds with higher write
            Engine: 'aurora-postgresql',
            DBInstanceClass: process.env.DBInstanceClass,
            EngineVersion: '14.6',
            PubliclyAccessible: false,
            AvailabilityZone: process.env.AvailabilityZone,
            MultiAZ: false,
            EnablePerformanceInsights: true,
            MonitoringInterval: 60,
            MonitoringRoleArn: process.env.MonitoringRoleArn
        };
        const data = await rds.createDBInstance(params).promise();
        console.log('Aurora instance created successfully:', data.DBInstance);
        return data.DBInstance;
    } catch (err) {
        console.error('Error creating Aurora instance:', err);
        throw err;
    }
};

CloudFormation Resources

Lambda function definition



  CreateRDSInstanceFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt AuroraScaleUpLambdaExecutionRole.Arn
      Runtime: nodejs16.x
      Timeout: 60
      Code:
        ZipFile: |
          const AWS = require('aws-sdk');
          const rds = new AWS.RDS();
          // ...
      Environment:
        Variables:
          DBClusterIdentifier:
            Ref: RDSCluster
          DBInstanceIdentifier: myapp-postgres-instance-3
          DBInstanceClass: db.r6g.xlarge
          AvailabilityZone: ap-northeast-1a
          MonitoringRoleArn: !Join
            - ""
            - - "arn:aws:iam::"
              - !Ref AWS::AccountId
              - ":role/rds-monitoring-role"

Scheduled rule to invoke the function before the peak



  ScheduledCreateRDSInstance:
    Type: AWS::Events::Rule
    Properties:
      ScheduleExpression: "cron(40 12 ? * MON-FRI *)"
      State: ENABLED
      Targets:
        - Arn: !GetAtt CreateRDSInstanceFunction.Arn
          Id: CreateRDSInstanceFunction

Scale up - Failover to newly created instance

Lambda function for failing over to new instance



const AWS = require('aws-sdk');
const rds = new AWS.RDS();
const logger = require('console');

const wait = (ms) => new Promise(resolve => setTimeout(resolve, ms));

const waitUntilRDSAvailable = async (dbInstanceIdentifier, maxAttempts = 90, pollingInterval = 10000) => {

    for (let attempt = 1; attempt <= maxAttempts; attempt++) {
      try {
          const response = await rds.describeDBInstances({ DBInstanceIdentifier: dbInstanceIdentifier }).promise();
          const dbInstance = response.DBInstances[0];
          const status = dbInstance.DBInstanceStatus;

          if (status === 'available') {
              console.log("RDS instance is available!");
              return;
          } else {
              console.log(`RDS instance is not yet available (Status: ${status}). Retrying...`);
              await wait(pollingInterval);
          }
      } catch (error) {
          console.error("Error describing RDS instance:", error);
          throw error; // Propagate the error to the caller
      }
    }

    console.error("Timeout: RDS instance did not become available within the specified time.");

    throw new Error("Timeout: RDS instance did not become available within the specified time.");
};

exports.handler = async (event, context) => {
  logger.log("Received event: ", JSON.stringify(event));
  const detail = event.detail;
  if (detail && detail.SourceType === "DB_INSTANCE" && detail.EventCategories.includes("creation") && detail.SourceIdentifier === process.env.DBInstanceIdentifier) {
      logger.log(`Received an instance creation event for ${process.env.DBInstanceIdentifier}`);
      const params = {
          DBClusterIdentifier: process.env.ClusterIdentifier,
          TargetDBInstanceIdentifier: process.env.DBInstanceIdentifier
      };
      try {
          await waitUntilRDSAvailable(process.env.DBInstanceIdentifier, 90, 10000); // 15 minutes
          try {
              await rds.failoverDBCluster(params).promise();
              console.log('Failover completed successfully');
          } catch (error) {
              console.error('Error during failover:', error);
              throw error;
          }
      } catch (error) {
          return { statusCode: 500, body: "Error waiting for RDS instance to become available: " + error.message };
      }
  } else {
      logger.log(`Received an event, but it is not an instance creation event for ${process.env.DBInstanceIdentifier}`);
  }
};

After an RDS creation event, RDS intance isn't immediately available. This is why we have to poll the status of the RDS instance periodically to check if it's avialable. Usually it will be avaiable after 2-5 minutes, but we set the Lambda timeout to 15 minutes (max) just to be safe.

I've also tried using the "availability" event, but unfortunately it's reserved for shutdowns and restarts. It won't be triggered if the RDS instance wasn't deleted or stopped recently.

CloudFormation resource

Lambda function definition



 RDSScaleUpFailoverFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt AuroraScaleUpLambdaExecutionRole.Arn
      Runtime: nodejs16.x
      Timeout: 900 # 15 minutes; wait for RDS instance to be available
      Code:
        ZipFile: |
          const AWS = require('aws-sdk');
          const rds = new AWS.RDS();
          // ...
      Environment:
        Variables:
          ClusterIdentifier:
            Ref: RDSCluster
          DBInstanceIdentifier: myapp-postgres-instance-3

EventBridge rule to trigger the lambda function in the event of instance creation



  RDSCreateDBInstanceEventRule:
    Type: "AWS::Events::Rule"
    Properties:
      Description: After DB instance created, invoke Lambda to failover RDS cluster
      EventPattern:
        source:
          - aws.rds
        detail-type:
          - RDS DB Instance Event
        detail:
          SourceIdentifier:
            - myapp-postgres-instance-3
          EventCategories:
            - creation
      State: ENABLED
      Targets:
        - Arn: !GetAtt RDSScaleUpFailoverFunction.Arn
          Id: RDSScaleUpFailoverTarget

Scheduled Scale-down for Aurora

Schedule a lambda function to run after write peaks.
The function fails over the cluster to the smaller instances.
On cluster failover, an RDS event triggers a Lambda function.
The function removes the large instance created during scale-up.

Scale down by failing over to smaller instance

Lambda function to fail over to smaller instance



const AWS = require('aws-sdk');
const rds = new AWS.RDS();

exports.handler = async () => {
  const params = {
      DBClusterIdentifier: process.env.ClusterIdentifier,
      TargetDBInstanceIdentifier: process.env.DBInstanceIdentifier
  };
  try {
      await rds.failoverDBCluster(params).promise();
      console.log('Failover completed successfully');
  } catch (error) {
      console.error('Error during failover:', error);
      throw error;
  }
};

CloudFormation

Lambda function definition



RDSScaleDownFailoverFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt AuroraScaleUpLambdaExecutionRole.Arn
      Runtime: nodejs16.x
      Timeout: 60
      Code:
        ZipFile: |
          const AWS = require('aws-sdk');
          const rds = new AWS.RDS();
          // ...
      Environment:
        Variables:
          ClusterIdentifier:
            Ref: RDSCluster
          DBInstanceIdentifier: myapp-postgres-instance-1

Scheduled rule to invoke the function after the peak



  ScheduledRDSScaleDownFailover:
    Type: AWS::Events::Rule
    Properties:
      ScheduleExpression: "cron(25 14 ? * MON-FRI *)"
      State: !FindInMap [EnvToParams, !Ref EnvironmentType, RDSDBScaleUpEnabled]
      Targets:
        - Arn: !GetAtt RDSScaleDownFailoverFunction.Arn
          Id: RDSScaleDownFailoverFunction

Remove created RDS in response to failover

Lambda function for removing the previously created instance



const AWS = require('aws-sdk');
const rds = new AWS.RDS();
const logger = require('console');

exports.handler = async (event, context) => {
  logger.log("Received event: ", JSON.stringify(event));
  const detail = event.detail;
  // ensure failover was for target instance
  if (detail && detail.SourceType === "CLUSTER" && detail.EventCategories.includes("failover") && detail.Message.includes(process.env.FailoverTargetDBInstanceIdentifier)) {
      logger.log(`Received an cluster failover event to ${process.env.FailoverTargetDBInstanceIdentifier}`);
      const params = {
          DBInstanceIdentifier: process.env.DBInstanceToRemoveIdentifier
      };
      try {
          await rds.deleteDBInstance(params).promise();
          console.log('Old replica instance deleted successfully');
      } catch (error) {
          console.error('Error during cleanup:', error);
          throw error;
      }
  } else {
      logger.log(`Received an event, but it is not an cluster failover event to ${process.env.FailoverTargetDBInstanceIdentifier}`);
  }
};

CloudFormation resources for removing RDS instance

Lambda Function definition



RemoveRDSInstanceFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt AuroraScaleUpLambdaExecutionRole.Arn
      Runtime: nodejs16.x
      Timeout: 60
      Code:
        ZipFile: |
          const AWS = require('aws-sdk');
          const rds = new AWS.RDS();
          // ...
      Environment:
        Variables:
          DBInstanceToRemoveIdentifier: myapp-postgres-instance-3
          FailoverTargetDBInstanceIdentifier: myapp-postgres-instance-1

EventBridge rule to trigger the lambda function in the event of cluster failover.



  RDSClusterFailoverEventRule:
    Type: "AWS::Events::Rule"
    Properties:
      Description: After RDS cluster failover to original reader, invoke Lambda to delete extra instance
      EventPattern:
        source:
          - aws.rds
        detail-type:
          - RDS DB Cluster Event
        detail:
          SourceIdentifier:
            - Ref: RDSCluster
          EventCategories:
            - failover
      State: ENABLED
      Targets:
        - Arn: !GetAtt RemoveRDSInstanceFunction.Arn
          Id: RDSScaleDownFailoverTarget

CloudFormation permissions and roles

Lambda Execution Role



  AuroraScaleUpLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: LambdaExecutionPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup # allow lambda to write to CloudWatch logs
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                  - iam:PassRole # allow lambda to pass rds-monitoring role to rds instance
                  - rds:DescribeDBInstances
                  - rds:FailoverDBCluster
                  - rds:CreateDBInstance
                  - rds:DeleteDBInstance
                Resource: "*"

Permissions for EventBridge Rule to invoke Lambda



InvokeCreateRDSInstancePermission:

  Type: "AWS::Lambda::Permission"

  Properties:

    Action: "lambda:InvokeFunction"

    FunctionName: !Ref CreateRDSInstanceFunction

    Principal: "events.amazonaws.com"

    SourceArn: !GetAtt ScheduledCreateRDSInstance.Arn

InvokeRDSScaleUpFailoverPermission:

  Type: "AWS::Lambda::Permission"

  Properties:

    Action: "lambda:InvokeFunction"

    FunctionName: !Ref RDSScaleUpFailoverFunction

    Principal: "events.amazonaws.com"

    SourceArn: !GetAtt RDSCreateDBInstanceEventRule.Arn

InvokeRDSScaleDownFailoverPermission:

  Type: "AWS::Lambda::Permission"

  Properties:

    Action: "lambda:InvokeFunction"

    FunctionName: !Ref RDSScaleDownFailoverFunction

    Principal: "events.amazonaws.com"

    SourceArn: !GetAtt ScheduledRDSScaleDownFailover.Arn

InvokeRemoveRDSInstancePermission:

  Type: "AWS::Lambda::Permission"

  Properties:

    Action: "lambda:InvokeFunction"

    FunctionName: !Ref RemoveRDSInstanceFunction

    Principal: "events.amazonaws.com"

    SourceArn: !GetAtt RDSClusterFailoverEventRule.Arn

Concerns

1. Downtime

The first worry with this approach is whether we will experience downtime with cluster failover. I ran some load testing, and it seems there's almost zero downtime with Aurora failover.

The load test is conducted with 10,000 requests, with failover being executed midway into the test. All requests went through successfully.

You can learn more about setting up your own load testing with k6 and EKS in my other blog post (Japanese only).

2. Excess replication lag

AWS recommends having the same size for your RDS instances. The reason is that weaker reader instance might not be able to catch up with the volume of changes in a stronger writer instance. This might cause the read replica to shut down when it falls behind the main writer instance too much.

My current setup single AZ with 2 db.t3.medium instances and 1 db.r6g.xlarge instance. I have monitored the AuroraReplicaLag in with spike tests, and it has never exceeded 85ms.

Shutdown issues with replica lag may occur after 60s, so be careful if you have a multi-AZ setup with hugely different instance sizes.

3. Data replication

One of my colleagues were concerned about possible slower startup times of instances once data accumulates, and that 30 minutes might not be enough for a new instance to spin up.

Unlike regular RDS, Aurora has a separate data layer that is synchronized across AZ.

While there may be cache residing on the instances, there is no need to replicate data across AZs when spinning up new intances. Therefore, time to spin up new intances in theory shouldn't be affected by the size of the data store.

After some testing wth inflating the database from 300Mb to 128GB, the startup time has remained consistently around 8 minutes.

Notes on automatic scale up

If write workloads are unpredictable, you can replace the EventBridge scheduled events with CloudWatch alarm triggers. That being said, starting a new aurora replica and failing over could take well over 10 minutes in my experience. With no AWS guarantee on how long it will take, it might not be a reactive approach, and the recommended way would just use a larger instance or use other DB services like DynamoDB or Redis on cluster mode instead.

References

Resolving next-auth "redirect" error when used with Supabase

Mathew Chan — Mon, 05 Feb 2024 04:59:19 +0000

Recently users on my side-project website have been experiencing login issues with oAuth (Discord, Twitch). Everything worked fine on localhost, but I would get Redirect error in production (hosted on vercel).

After many hours of trying out different configurations and looking through next-auth Github issues, I decided to upgrade next-auth from v3 to v4. The migration guide is here.

There are many bumps when upgrading, including releasing foreign keys from users table, and reverting migration changes recommended by the migration guide.

-- sessions.expires has to remain of type TIMESTAMPTZ instead of TEXT
-- don't run this line when migrating your postgres table
ALTER TABLE sessions ALTER COLUMN "expires" TYPE TEXT USING CAST(CAST(extract(epoch FROM "expires") AS BIGINT)*1000 AS TEXT);

To my dismay, however, that didn't resolve the issue.

I recalled that Supabase was dropping IPv4 support and moving to IPv6. Apparently, with that change, they are also dropping PGBouncer.

The solution was to change the connection string to avoid using PGBouncer.

# .env.local
DATABASE=postgres://postgres:[password]@db.[project-ref].supabase.co:5432/postgres
## DATABASE=postgres://postgres:[password]@aws-0-us-east-1.pooler.supabase.com:5432/postgres?options=reference%3D[project-ref]

And then update next-auth config pages/api/auth/[...nextauth].js to use the connection string.

import NextAuth from 'next-auth'
import DiscordProvider from "next-auth/providers/discord";
import TwitchProvider from "next-auth/providers/twitch";
import PostgresAdapter from "@auth/pg-adapter"
import { Pool } from 'pg'

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 20,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
  ssl: {
    rejectUnauthorized: false
  }
})

export default NextAuth({
  // Configure one or more authentication providers
  providers: [
    DiscordProvider({
      clientId: process.env.DISCORD_CLIENT_ID,
      clientSecret: process.env.DISCORD_CLIENT_SECRET
    }),
    TwitchProvider({
      clientId: process.env.TWITCH_CLIENT_ID,
      clientSecret: process.env.TWITCH_CLIENT_SECRET
    })
    // ...add more providers here
  ],
  secret: process.env.NEXTAUTH_SECRET,
  callbacks: {
    session: async (session, user) => {
      return session;
    }
  },
  pages: {
    signIn: '/auth/signin',
  },

  adapter: PostgresAdapter(pool),
})

Also remember to include a next-auth secret when upgrading to version 4 of next-auth.

Version 5 is already in beta of next-auth, but this upgrading experience has definitely made me rethink about making any breaking changes to my database.

Python Text Parsing Project: Furigana Inserter for Anki

Mathew Chan — Tue, 31 Aug 2021 19:57:50 +0000

Furigana

Furigana is a Japanese reading aid for Kanji, which originally came from Chinese.

There are many implementations like this for inserting Furigana into Japanese text on the web through ruby formatting, but in this post we will cover the Anki format.

Anki's Furigana Format

Anki's Furigana format is more readable and shorter, making editing easier as well as saving disk space. It's used on Anki but it's quite straightforward to convert it to the ruby format for the web.

Here are the examples for 辞やめたい and 辿たどり着つく:

Ruby format: <ruby><rb>辞</rb><rt>や</rt>めたい</ruby>
Anki format:　辞[や]めたい

Ruby format: <ruby><rb>辿</rb><rt>たど</rt>り</ruby><ruby><rb>着</rb><rt>つ</rt>く</ruby>
Anki format:　辿[たど]り着[つ]く

Currently the two predominant tools for inserting Furigana on Anki are the Japanese Support addon and the Migaku Japanese addon.

There are some issues with these tools, however, as they fail on one or more of these issues.

Hiragana in the middle of the word
Special Japanese characters and punctuation parsing
User-defined mapping for specific words with multiple readings as well as names and places.

We will address these issues in this project.

Sudachi for Text Segmentation

Text segmentation is a method to break down a sentence into smaller chunks called morphemes. This will enable us to look up the individual readings of each morpheme and apply a Furigana reading when necessary.

Instead of the common segmentation tool Mecab, this project will use Sudachi, which features multiple text segmentation modes as well as Furigana retrieval.

We will also use the Wanakana package for Python to for basic Japanese text manipulation.



pip install sudachipy sudachidict_small wanakana-python

Sudachi Segmentation Modes

There are three segmentation modes in Sudachi - A, B, and C.

We will use mode C to get the reading of the longest possible entries in the sentence.



from sudachipy import tokenizer
from sudachipy import dictionary

tokenizer_obj = dictionary.Dictionary(dict_type="small").create()

def add_furigana(text):
    tokens = [m for m in tokenizer_obj.tokenize(text, tokenizer.Tokenizer.SplitMode.C)]
    parsed = ''
    # ...
    return parsed

Skip Punctuation and Special Characters

We also need to define certain non-Japanese characters (but considered as Japanese symbols by the Wanakana library:



from wanakana import to_hiragana, is_japanese, is_katakana, is_hiragana
import string

JAPANESE_PUNCTUATION = '　〜！？。、（）：「」『』０１２３４５６７８９ａｂｃｄｅｆｇｈｉｊｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ'

def is_japanese_extended(text):
    return is_japanese(text) and text not in string.punctuation and text not in JAPANESE_PUNCTUATION

We can then add the logic to our furigana insertion method. to_parse is only True if the (morpheme) token contains Japanese characters with Kanji. In other words it is False if the text is purely Hiragana or Katakana.

Special characters like 〜 have to be singled out as they are attached to tokens in Sudachi.




SPECIAL_CHARACTERS = '〜'

def add_furigana(text):
    tokens = [m for m in tokenizer_obj.tokenize(text, tokenizer.Tokenizer.SplitMode.C)]
    parsed = ''
    for index, token in enumerate(tokens):   
        to_parse = is_japanese_extended(token.surface()) and not is_katakana(token.surface()) and not is_hiragana(token.surface())
        if to_parse:
            if token.surface()[-1] in SPECIAL_CHARACTERS:
                parsed += add_furigana(token.surface()[:-1]) + token.surface()[-1]
            else:
                if index > 0:
                    parsed += ' '
                  reading = to_hiragana(token.reading_form())
                # ...
         else:
             parsed += token.surface()
     return parsed

There are multiple methods in the token item. We use the surface() method to get the original string, and reading_form() to get the Furigana reading of the morpheme.

An important note is that the reading_form() method returns the reading in Katakana, which needs to be converted to Hiragana and can be achieved with Wanakana's to_hiragana() method.

User-defined Mapping and Parsing

We can create a small dictionary to map custom items we want to do our own mapping. This can be everything from using more common readings (私 returns the reading わたくし instead of わたし) to names like 優那[ゆうな] which Sudachi incorrectly parses to 優[ゆう] 那[].



KANJI_READING_MAPPING = {
    '私': '私[わたし]',
    '貴女': '貴女[あなた]',
    '何が': '何[なに]が',
    '何を': '何[なに]を',
    '我国': '我国[わがくに]',
    '行き来': '行[い]き 来[き]',
    '外宇宙': '外宇宙[がいうちゅう]',
    '異星人': '異星人[いせいじん]',
    '優那': '優那[ゆうな]',
    '菜々美': '菜々美[ななみ]'
}

We can also use a JSON file for easier data management but here we will keep it contained in the Python file.

To map them manually we check if their surface form is one of the keys of our KANJI_READING_MAPPING and append the corresponding value to our parsed result.

Sometimes the keyword may cover 2 tokens, for example 異星人 are broken down into the tokens 異星[いせい] and 人[にん], and 優那 into the tokens 優[ゆう] and 那[]. This is why we also check to see if the combined term of the current token and the next token token.surface() + tokens[index+1].surface() are also a key in our mapping dictionary.



token_indexes_to_skip = []
for index, token in enumerate(tokens): 
    if index in token_indexes_to_skip:
        continue
    # ...
    if index < len(tokens)-1 and token.surface() + tokens[index+1].surface() in KANJI_READING_MAPPING:
        parsed += KANJI_READING_MAPPING[tokens[index].surface() + tokens[index+1].surface()]
        token_indexes_to_skip.append(index+1)
    elif token.surface() in KANJI_READING_MAPPING:
        parsed += KANJI_READING_MAPPING[token.surface()]
    else:
    # ...

We add a list token_indexes_to_skip to store the index+1 tokens that are already used for the custom mapping. This way we can skip parsing it in the next iteration.

Hiragana in the middle of the word

We set two pointers, surface_index to go through each character of the surface form, and reading_index for the *reading.

For example, the surface form 教える has the reading form オシエル and subsequently おしえる after Hiragana conversion.

If the character is Hiragana or Katakana, we directly add it to the parsed string and increment both the pointers.

If it is Kanji, we find the reading for the Kanji by using a temporary variable reading_index_tail and increment that until we reach the end of the reading string or until the next reading character matches the next Hiragana character in the surface string.

By matching え in おしえる to 教える we can parse the reading of 教 to おし. reading_index's value is 0 and reading_index_tail's value is 1

We will also add an extra condition to move the reading pointer if the subsequent Hiragana character is the same as the current one.

For example 可愛い is parsed as 可愛[かわい]い instead of 可愛[かわ]いい. By checking the repetition of the final い we could force the first い to be included in the reading.



surface_index = 0
reading_index = 0
while len(token.surface()) > surface_index:
    if is_hiragana(token.surface()[surface_index]) or is_katakana(token.surface()[surface_index]):
        parsed += token.surface()[surface_index]
        reading_index += 1
        surface_index += 1
    else:
        next_index = -1
        for token_index in range(surface_index, len(token.surface())):
            if is_hiragana(token.surface()[token_index]) or is_katakana(token.surface()[token_index]):
                next_index = token_index
                break
            if next_index < 0:
                parsed += to_anki_format(
index=surface_index, kanji=token.surface()[surface_index:], 
    reading=reading[reading_index:])
                break
            else:
                reading_index_tail = reading_index
                while reading[reading_index_tail] != token.surface()[next_index] or (reading_index_tail < len(reading)-1 and reading[reading_index_tail] == reading[reading_index_tail+1]):
                    reading_index_tail += 1
                parsed += to_anki_format(
                                  index = surface_index, 
                                  kanji = token.surface()[surface_index:next_index], 
                                  reading = reading[reading_index:reading_index_tail])
                reading_index = reading_index_tail
            reading_length = next_index - surface_index
            if reading_length > 0:
                surface_index += reading_length
            else:
                break

Anki Format

Anki format is pretty straightforward. We add an extra space in the beginning if the parsed Kanji is in the middle of the sentence.



def to_anki_format(index, kanji, reading):
    return '{}{}[{}]'.format(' ' if index > 0 else '', kanji, reading)

Putting it together

There has been a lot more steps and conditional checking but in the end our parser could handle a lot more cases and be highly customizable.

Note: to run this example, you may need to run pip install sudachidict_small in the repl shell to install the Sudachi dictionary.

Extra

Anki

So far we have covered how to add Furigana to Japanese text in the Anki format but we have not covered how to make an Anki addon (plugin). It could be a simple hotkey control to convert a field to Japanese with Furigana, or it could be an addon to batch insert Furigana to an existing Anki Deck. We will leave that for another post.

Web

Assume you have a relational database of Japanese text and you added an extra column to store all the text with inserted Furigana. You would need to retrieve the data and show it on the Web in ruby format.

You will need to write a simple text formatting function. No need for segmentation libraries here. You can have a look at my React JS implementation if you need help.

Budget Cloud Resources for Hobbyists

Mathew Chan — Sat, 07 Aug 2021 11:02:35 +0000

Background

I'm currently running the following projects:

Python Twitch Bot
Python Flask Webapp to provide full-text search for 200k documents
Next JS Frontend that consumes the API from my web apps
Desktop App Documentation Website
NocoDB as CMS for document and media collaboration

There are many budget cloud services out there, but I decided to go with the ones that are easy to use and have predictable pricing. After all I'm interested in mostly developing my apps, not maintaining servers or managing Kubernetes.

Disclaimer: I'm not affiliated with or sponsored by any companies mentioned in this post.

Python Anywhere (PA)

This is where I host my Python webapps. The biggest draw is its ease of use but for me the most important thing is 3 GB RAM even for the cheapest Hacker plan ($5/month).

If you contact them through email, you can even pay upfront yearly to have 2 months off (which translates to ~$4.2/month!)

Other Pros:

No hard limit on data outbound/inbound
Almost everything you need for your Python work including notebooks, online consoles, background tasks, and server logs.

Cons:

Have to wait for PA (Python Anywhere) to update their Disk image for Python dependencies.
No support for Starlette and Fast API at the time of this writing
Hacker plan only offers 1GB of storage. For an extra dollar/month you can get 4GB of extra storage. Storage seems to be the most expensive on this platform.

Workflow

After I select the web app template (Flask/Django/Bottle), I just add a git remote to Github/Bitbucket and I can directly pull the changes from my repo to the server. I still have to press the restart server on the website or use their API after I git pull.

Digital Ocean (DO)

On the other hand, I can configure DO instances with any language or framework using Docker containers. This is where I plan to host my Fast API server in the coming future.

Pricing starts at $5/month as well with 25 GB of storage but only 1 GB of RAM. Storage is a lot cheaper on DO but RAM is expensive ($10/m for 2 GB and $20/m for 4 GB) so depending on how much memory and storage your app needs you should decide whether you want to host on PA or DO.

Workflow

Setting up servers on DO takes a bit more time with proxies, domains, and SSH certificates but Dokku makes the process much faster.

Also, with Dokku I can set up git-based deployments so I can directly push changes to the server and trigger a redeploy making it an ever faster experience similar to Heroku.

Digital Ocean Spaces

I use Digital Ocean Spaces for my storage at $5/month for 250 GB.

There are free storage options out there such as Scaleway (75 GB free!) but I haven't tried that out yet.

The CDN in DO Spaces is fast and using an S3 client like Cyberduck makes it easy to manage my media files.

Netlify

This is where I host my app documentation website generated with MkDocs for free. Setting up the build command for the MkDocs site can be done by just adding a netlify.toml config file with three lines.



[build]
  command = "mkdocs build"
  publish = "site"

Git-based Workflow

I only need to select my Github/Bitbucket repo on the platform and I can trigger builds by pushes to any branch.

Vercel

This is where I host my Next JS app that consumes the Python API. Generous serverless function quotas and native support for Next JS apps here is great.

I've tried hosting Next JS apps on Netlify but there are some issues with client side fetching when reloading web pages. Maybe this will be resolved in the future but it will always lag behind Vercel when it comes to deploying Next JS apps.

Git-based Workflow

This works the same as Netlify: select my Github/Bitbucket repo on the platform and trigger redeploys by git pushes.

Railway

Railway is a new cloud provider. They've changed their pricing recently but their template apps are super easy to deploy with a single click.

I'm hosting a NocoDB with a PostgreSQL database as well as an Umami analytics platform (with another Postgres DB) for my website.

Adding self-hosted analytics to your website for free with Umami

Mathew Chan ・ Aug 5 '21

#webdev #tutorial

Pricing is based on usage, but there is a $5 free quota which is plenty for a single small app. I also found it to be the cheapest way to host a Redis store as managed Redis on even DO starts at $15/month with only 0.5 GB of RAM...

At the time of this writing, I receive a daily email of how much they estimate I would need to pay this month as well as how much I've spent so far. I can also check the numbers on my dashboard so that's reassuring. Service stops once the credit is exhausted which is not ideal for mission critical services, but for me I prefer that over getting hefty and shocking cloud bills.

What I have stopped using

MongoDB Atlas

After mLab was purchased by MongoDB Cloud, I had to switch my Mongo database to Atlas and the connection of my Python app seems to keep getting timed out because it was a shared cluster which never happened when I was using mLab.

I think I will just use Railway to host my Mongo databases for the time being.

Ending Thoughts

Cloud bills can easily rack up and having read horror stories of people's AWS or GCP bills, I've decided to go with monthly/annual plans, with Railway being the only exception for now.

Adding self-hosted analytics to your website for free with Umami

Mathew Chan — Thu, 05 Aug 2021 07:06:00 +0000

Background

Google Analytics is free but there have been privacy concerns over the handling of user data. Zoom was recently fined 85 million for misleading users about their Google and Facebook trackers.

With ad blocks and tracking prevention built right into browsers like Safari, it is time to consider hosting your own analytics platform.

Start an Umami Project on Railway

Umami is free, open source, and dead simple. It requires a server with Node JS and either a MySQL or PostgreSQL database.

You can set that up with a single click on Railway.

You can use my referral link to register and get a $5 credit. You also have a monthly $5 quota of free cloud resources and credit card not required. This should be plenty for Umami.

Go start a new project and select Deploy Starter.

Select Umami.

Ackee is another open source analytics platform you can deploy on Railway. I first tried Ackee but I prefer Umami because of its Realtime Dashboard that allows me to see active sessions and what webpages are being visited at this very moment.

I hate to admit it but I might be a bit addicted to seeing what people are looking at on my website 😅 so I try not to visit my Umami dashboard that often.

Select your Github account and name the repository for Umami.

You will need to enter HASH_SALT which can be easily generated on the platform with CMD+K -> Generate.

Once you click deploy, both the Node server and the Postgres database will be set up.

Once they have been deployed, click on the Deployments tab and click on the link below the Success message to go to the Umami platform.

🎉🎉 Congratulations! Your own analytics platform 📈 is set up!

Login

The default account has the username admin and the password umami.

Add a website

In Settings, navigate to Websites and click on the Add Website button.

From Umami:

The Name field can be whatever you want. Usually it's the same as the domain name.

The Domain field is the actual domain of your website. It is used to filter out your own website from the list of referrers in your metrics.

The Enable share URL checkbox means you want to share your website stats via a unique URL.

After a website is added, we can get its tracking code.

A form will pop up with a code containing a script tag. You can then insert it to the <head> section of your website.

Next JS app

If you are adding it to your Next JS app, you can add it to your _app.js file and enclose it inside next/head. You can also use the environment variable NODE_ENV to render the tracker only on production environments so it won't collect data on your local environment while you're developing the website.



import Head from 'next/head';

function MyApp({ Component, pageProps }) {
  return (
    <>
      {process.env.NODE_ENV === 'production' &&
        <Head>
          <script async defer data-website-id="4fb7fa4c-5b46-438d-94b3-3a8fb9bc2e8b" src="https://your-umami-app.com/umami.js"></script>
        </Head>
      }
      <Component {...pageProps} />
    </>
  )
}

export default MyApp

Enjoy your new analytics portal!

No Social Media trackers!

You can verify that Umami does not show as a tracker on browsers and security extensions.

Since it doesn't use cookies, there is no need to show cookie consent pop up forms.

Safari Tracker Report:

DuckDuckGo Privacy Essentials:

Migrating your Next.js app from Vercel to Netlify

Mathew Chan — Tue, 29 Jun 2021 10:24:20 +0000

I recently moved my website https://www.immersionkit.com from Vercel to Netlify because I reached the image edge optimization limit on Vercel.

Background

Apparently, there is a limit of 1000 image optimization processes per month on Vercel, something that is counted whenever you access an image being handled by next/image. You can't see your usage currently, so you won't know when you have reached your limit until they shut down your account and send you a friendly email.

After a short email exchange I deleted the app on Vercel and they nicely reactivated my account.

~~From my experience so far, such limits aren't imposed on Netlify and their image edge caching is superb.~~ Image caching is counted as functions and your function usage is shown in Site Settings > Functions. If you exceed the usage you will be upgraded to Tier 1 ($25/monthly).

There are, however, a few extra steps to configure your app on Netlify for a project that was previously hosted on Vercel.

1. New site from git

This works similarly to Vercel and according to my experience, with even less hiccups. You simply choose the repository of your website, use

next build

for the build command, and Netlify will automatically build the source from the master branch and deploy the site.

Remember to set the environment variables if there are any.

Unlike Vercel, Netlify won't automatically deploy the other branches so if you need other branches like develop or staging to be deployed you need to update your branch deploys to All in Deploy Contexts.

2. Essential Next.js Plugin

Server side rendering and image edge caching are handled differently on Netlify. To enable these features we need to add the Essential Next.js plugin.

SSR lets you use functions like getServerSideProps while image edge caching is required for next/image

When I deployed my application it asked me if I wanted to install the plugin. If you didn't get the prompt you can add the plugin manually in the Plugins tab.

3. Redirects and rewrites

Redirects on Vercel are defined in the next.config.js file but on Netlify in the netlify.toml file.

The netlify.toml file is a configuration file that specifies how Netlify builds and deploys your site — including redirects, branch and context-specific settings, and more. - Netlify

This is what my redirects look like in my next.config.js.

async redirects() {
    return [
      {
        source: '/home',
        destination: '/',
        permanent: true,
      },
      {
        source: '/dictionary/:slug',
        destination: '/dictionary/?keyword=:slug',
        permanent: true,
      }
    ]
  },

Let's configure these redirects on Netlify.

First create a netlify.toml file and add a redirect like this.

[[redirects]]
  from = "/home"
  to = "/"
  status = 301

301 redirects are equivalent to permanent redirects on Vercel.

On Netlify, instead of :slug, we use the asterisk * and :splat.

[[redirects]]
  from = "/dictionary/*"
  to = "/dictionary?keyword=:splat"
  status = 301

Push your config file to the master branch and trigger a re-deploy. The redirect links should work now.

4. Asset Optimization

In build settings you can also minimize your asset files for your website to be served quicker. The difference can be pretty significant on mobile devices.

Conclusion

Migrating your site to Netlify should be pretty straightforward but there are a few tweaks you have to make. I think it's worth it overall.

The biggest downside on Netlify for me is that you're limited to 300 build minutes per month on the free plan with no logging or analytics available. There are some analytics on Vercel but that too is fairly limited so if you need analytics it's better to use a service like Google Analytics or Fathom.

Article No Longer Available

Quick way to serve static files with Netlify (100GB free)

Mathew Chan — Tue, 08 Jun 2021 14:01:30 +0000

Needed to publish a website and serve a few GB of images and audio data. Here are my findings and what I ended up doing.

Free Options

There are storage and bandwidth limits on many common sites. Bandwidth limit is how many times users can access your files per month.

Github Pages (1GB limit)
Bitbucket Cloud (2GB limit)
Qovery (10GB limit)
Cloudinary (25GB storage and 25GB bandwidth/month)
Vercel (100GB storage and 100GB bandwidth)
Netlify (100GB storage and 100GB bandwidth)

Paid options

AWS S3
Google Cloud Storage
Digital Ocean Block Storage

These storage options (especially for fast CDN serving) can run up to $20/month for just 10-20GB of data.

Netlify & Vercel

Both Netlify & Vercel offer 10 GB of free storage and 10GB of bandwidth/month. I already use Vercel, but I since I have to use github or bitbucket to upload my files I would still hit the storage limits of the repository platforms.

Netlify Large Media

For Netlify, there is a Large Media to use Git LFS to basically upload the files to Netlify and put pointer files on your Github repository.

After hours of tinkering I still couldn't upload files to Netlify and the workflow was awful.

Netlify sync with Publii

Surfing the Internet I found a desktop website builder called Publii that can sync to Netlify.

It was super easy to set up and after setting up the API key, I just need to press the Sync button to upload all my media files in the media folder of the preset template.

There is no version management because git is not setup but I prefer this workflow just to serve my jpg and mp3 files. Obviously there are ways to upload my files through Netlify CLI but this app just makes it much easier to set up.

3 Projects I Plan to Release This Year

Mathew Chan — Wed, 02 Jun 2021 18:28:02 +0000

Introduction

I plan to release 3 side projects to help people who are learning Japanese. They are mostly inspired by the my first project and the feedback I received.

What I have done so far

My original project helped language learners to extract text, look up words, and create sentence cards in Anki all in one app. It was OCR, dictionary, media recorder, script matcher, text hooker, and Anki integration all in one easy-to-use app.

This inspired me of many new ideas, but on the macro-level I begin thinking of apps that could reduce the friction of online learning.

The State of Language Learning

In the current state, language learners go through 3 stages.

Stage I: Curation: They look for content through recommendation lists or study guides. They may or may not be suitable for their level.

Stage II: Material: This is where they enjoy the content. It could be a movie on Netflix, a game on Steam, or simply a physical book.

Stage III: Tools: They download tools, dictionaries, and programs to facilitate learning through the material in Stage II.

Most stuff you would find belongs to either of these lists. There are some websites that do a pretty good job of having two of these elements, like bilingual manga, animelon, and Satori reader.

For my newer projects, I plan to include ALL of these stages and reduce friction as much as possible for newcomers.

Project I: Yuzu

This is an app for educators to easily turn native content into language learning material. An educator could take a book, a news article, or manga and add highlightable text to it with audio, translation, teacher's notes.

Highlightable text is super important in Japanese learning because it allows the user to look up words immediately with a popup dictionary instead of having to write unfamiliar Kanji or use image recognition software.

I also want to release a website with sample learning material curated by learner level tags such as WK level/JLPT level. The material will initially be public domain or CC content such as works of children literature.

Project II: Word List to Ank (web app)

Many language learners already have a list of words that they want to memorize. This converts them to premade Anki cards embedded with sentences and audio from anime/manga/TV shows. Users can also select the type of card they want for different studying purposes like listening purpose or optional recall (recalling the Japanese phrase when presented with the English definition).

Project III: Twitch Extension for Streamers Who Teach Japanese

There have been more Twitch streamers who are teaching Japanese as they play Japanese games. Streamers like Japanese Quest and Game Grammar are sharing how fun learning languages can be while playing games.

Although I admit that I've always be skeptical of how effective learning languages through games or through someone's stream can be, I can't deny that there a growing number of people who are interested in learning Japanese because of these streams.

For this project I plan to use Twitch's extension feature to let viewers interact with the stream content to learn Japanese or contribute. The most obvious feature would be to show the game text on the screen like subtitles so viewers can use popup dictionaries or look those words up while watching the stream, but with digital tools we should be able to do much more than that.

Quizes, adding words to a word list, bookmarking words, using an embedded dictionary are all things that come to mind.

But...what about games? Or something that we don't usually do in traditional classroom? Since we are not looking for effectiveness, digital tools should in theory pave way for more innovative ways to learn.

Perhaps Shiritori? Bingo? Not really related to the stream.

Memory cards? Not that fun for mature learners.

I went over the Internet and found this simple little game called Kanji Press.

Three things I like about this game.

simplicity
requires a certain degree of input from the user while not requiring any typing
can work with vocabulary in the stream

I might experiment with this game or do something completely different.

Let me know if you have good ideas.

Making a Japanese Dictionary Lookup Tool with Sudachi in Python

Mathew Chan — Mon, 26 Apr 2021 12:49:56 +0000

Background

I've been integrating a dictionary lookup tool in my recent project to help others learn words from games. The following are my goals.

Goals

These goals define user input and the desired output.

1. Get longest possible entry of user input

Input: 自己紹介（じこしょうかい）
Desired Output: 自己紹介, not 自己 or 自

2. Context-unaware

Input: 牛 in 牛丼を食べてる
Desired output: 牛(うし), but not ぎゅう as in ぎゅうどん（牛丼）

3. Parse conjugated verbs and adjectives

Input 食べて in 牛丼を食べてる
Desired Output: 食べる（たべる）

4. Dictionary used in Parser independent from dictionary used for the entry's glossary.

(More later.)

Our desired output assumes a certain degree of Japanese grammatical knowledge from the user. When the user selects 折 from 折角 we assume they want to know what 折 means instead of 折角. When they select 外国人 they want to look up the entire compound noun instead of its individual characters.

Setting up

Dictionary - JMDict

We will be using JMDict, a freely available Japanese-to-English library. You can find Japanese to other language libraries on the Yomichan project.

from pathlib import Path
import zipfile
import json

SCRIPT_DIR = Path(__file__).parent 
dictionary_map = {}

def load_dictionary(dictionary):
    output_map = {}
    archive = zipfile.ZipFile(dictionary, 'r')

    result = list()
    for file in archive.namelist():
        if file.startswith('term'):
            with archive.open(file) as f:
                data = f.read()  
                d = json.loads(data.decode("utf-8"))
                result.extend(d)

    for entry in result:
        if (entry[0] in output_map):
            output_map[entry[0]].append(entry) 
        else:
            output_map[entry[0]] = [entry] # Using headword as key for finding the dictionary entry
    return output_map

def setup():
    global dictionary_map 
    load_dictionary(str(Path(SCRIPT_DIR, 'dictionaries', 'jmdict_english.zip')))

To load our dictionary, we unzip the file and save it as a map with all its entries as the keys. We also check entries with repeat glossaries and add them to its list of glossaries.

Parser - Sudachi

pip install sudachipy
pip install sudachidict_small

To save space, we will use Sudachi's small dictionary instead of its core (70Mb).

from sudachipy import tokenizer
from sudachipy import dictionary

tokenizer_obj = dictionary.Dictionary(dict_type='small').create()
mode = tokenizer.Tokenizer.SplitMode.A

There are three modes in Sudachi - A, B, and C. Mode A parses words in its longest possible form where C its shortest. For our use case we will stick to mode A since we want the longest.

Putting it together

def look_up(word):
    word = word.strip()
    if word not in dictionary_map:
        m = tokenizer_obj.tokenize(word, mode)[0]
        word = m.dictionary_form()
        if word not in dictionary_map:
            return None
    result = [{
        'headword': entry[0],
        'reading': entry[1],
        'tags': entry[2],
        'glossary_list': entry[5],
        'sequence': entry[6]
    } for entry in dictionary_map[word]]
    return result

We first remove any unnecessary white spaces around our word then we directly check if it exists in our dictionary. This way we can get nouns like 牛丼 immediately without having to parse them.

After that we parse them with Sudachi mode A and get the dictionary_form() of the word and look that up in our own dictionary instead of using the parser's dictionary.

The final result is reformatted and returned.

(env) $ python
>>> setup()
>>> print(look_up('牛丼'))
[{'headword': '牛丼', 'reading': 'ぎゅうどん', 'tags': 'n', 'glossary_list': ['rice covered with beef and vegetables'], 'sequence': 1845250}]
>>> print(look_up('食べて'))
[{'headword': '食べる', 'reading': 'たべる', 'tags': 'v1 vt', 'glossary_list': ['to eat'], 'sequence': 1358280}, {'headword': '食べる', 'reading': 'たべ
る', 'tags': 'v1 vt', 'glossary_list': ['to live on (e.g. a salary)', 'to live off', 'to subsist on'], 'sequence': 1358280}]
>>> print(look_up('自己紹介'))
[{'headword': '自己紹介', 'reading': 'じこしょうかい', 'tags': 'n vs', 'glossary_list': ['self-introduction'], 'sequence': 1317650}]

Let me know if this was helpful.

Playing MP3 Files in Python with Pydub and Pyaudio

Mathew Chan — Thu, 15 Apr 2021 07:21:01 +0000

Introduction

Pyaudio allows us to play and record sounds with Python. To play MP3, however, we first need to convert the MP3 file to WAV format with ffmpeg. To use ffmpeg in Python, we use an interface tool called Pydub, which directly calls our ffmpeg executable and integrates with Pyaudio.

Installation

pip install pyaudio
pip install pydub

Playing sound

from pydub import AudioSegment

song = AudioSegment.from_mp3('test.mp3')
play(song)

If ffmpeg isn't installed on the machine, AudioSegment will fail to locate the mp3 file. There are a few solutions:

Install ffmpeg and add to environment path
Copy ffmpeg to the same folder as the python file
Static link ffmpeg to pydub

Let's talk about the third option.

Static linking ffmpeg

By static linking ffmpeg, our program would be portable: users would not need to manually install ffmpeg to their OS.

Download ffmpeg and copy it to your script directory.

import os
import platform
from pathlib import Path
from pydub import AudioSegment

AudioSegment.ffmpeg = path_to_ffmpeg()
platform_name = platform.system()
if platform_name == 'Windows':
    os.environ["PATH"] += os.pathsep + str(Path(path_to_ffmpeg()).parent)
else:
    os.environ["LD_LIBRARY_PATH"] += ":" + str(Path(path_to_ffmpeg()).parent)

def path_to_ffmpeg():
    SCRIPT_DIR = Path(__file__).parent 
    if platform_name == 'Windows':
        return str(Path(SCRIPT_DIR, "win", "ffmpeg", "ffmpeg.exe"))
    elif platform_name == 'Darwin':
        return str(Path(SCRIPT_DIR, "mac", "ffmpeg", "ffmpeg"))
    else:
        return str(Path(SCRIPT_DIR, "linux", "ffmpeg", "ffmpeg"))

song = AudioSegment.from_mp3('test')
play(song)

The program should now be able to run and play the mp3 file. The path added to os environment is temporary and would be gone when the python program is completed.

A Summary on React 2025 - a Modern SaaS Course

Mathew Chan — Sun, 11 Apr 2021 19:09:55 +0000

While I was looking for portfolio templates online, I found a sleek looking web portfolio by Lee Robinson and there I found his free course React 2025 on building a modern SaaS application. Here are my takeaways of the course. I am not affiliated with Lee in any way nor do I receive anything for writing this summary.

What the Course is About

React 2025 consists of a series of hour-long videos where he goes through different elements that form a SaaS - design, authentication, integrating payment, and testing.

Videos are of high quality covering how he builds the application from scratch to finish. Sometimes he gets stuck but he demonstrates how he can debug his way out. I didn't follow step by step to build the same application, but there were many things I learned in this course.

What We build

Fast Feedback - an SaaS application that lets users to easily add reviews and comments to their site through an iframe tag and url.

Design Process

Excallidraw -> Figma -> OpenChakra.
Course shows how you can convert rough drafts in Excallidraw to finer designs in Figma, and finally move Chakra UI components in OpenChakra.

Excallidraw

Figma

OpenChakra

Having tried out Excallidraw, I feel like I could easily and qiuckly do the same kind of draft work by pen and paper, but it does look a bit more professional if I had to present this design to someone else, say, a professional designer who can turn that into a more detailed design document.

I've heard of Figma but never used it, but since OpenChakra lets you drag and drop components into a design sheet, I see no need to use Figma if we just dive straight with OpenChakra.

Chakra UI

Component library that lets users create nice-looking apps easily. "Style props" that makes you modify styles on your components for good JSX readability, but at the same time extra style names to learn and not a must imo if you're already familiar with CSS.

Domain

Lee recommends using Vercel for buying domains. Not sure if his recommendation is partly because he works for Vercel now but when I checked domain prices on Vercel and it was $20/year for unclaimed .com domains! That's over double the price you can get on other websites like Namecheap and Google Domains. And don't forget Google Domains offer free analytics tools, which over on Vercel demands an extra $20/month subscription.

Instant Deployment with Vercel

Quick deployment helps you prototype and show your work to other people. The course demonstrates it's simply pushing your branch to master and letting Vercel do the rest.

Database

Firebase is used for handling all data storage. To be honest, I've never been a fan of firebase and I prefer mongoDB, but the course really persuaded me how easy and intuitive it can be to prototype and scale your app with Firebase.

There is a separate tab for 3rd Party Authentication on the Firebase dashboard so you can easily add Github, Twitter, or Facebook authentication to your SaaS application. It looked really convenient to set things up and you can get authentication on your website in half a day.

React with Next JS

React Hooks make it even easier by createContext() in your auth file and exporting it as a hook that can be reused in consumed by your other pages that needs to show authentication buttons and user information.

We are introduced to different ways of rendering websites - client side, server side, and static site generation.

There's also a concept called ISR (incremental static regeneration) where we can override the getStaticProps function so certain data in our statically generated pages can be checked every so often and updated.

We can also fetch data on client side (during build time) with a React Hook library called SWR. It's basically a wrapper around Fetch.

This video explains this best.

React Hook Forms: new Forms library. Supposedly better performance because of less re-rendering.

Stripe Payments

Stripe is used for development and it requires a bit of effort to set things up between Stripe, your app, and Firebase auth. Lee ran into some issue demonstrating how to set things up but in the end he managed to debug it.

Miscellaneous

Adding Terms of Use and Privacy, Adding meta tags to improve SEO, logging with pino, testing with puppeteer, and using fathom instead of Google Analytics for better customer privacy.

Marketing

Lee doesn't claim to be an expert on marketing and distribution, but he briefly mentioned that images and videos are important, as well as having vibrant colors on your website. General advice on posting your app on websites where your users are, and that you can write about your journey, why you developed it (how's it different from current solutions), and how you developed it.

Conclusion

I enjoyed the course a lot. Many things I would do differently but a lot of takeaways and things to use when starting my next web project.

Making Tesseract Portable in MacOS with Runtime Linking

Mathew Chan — Wed, 07 Apr 2021 16:38:34 +0000

Background

I like the plug and play experience. Recently, I developed a desktop app for people to learn Japanese, but it required MacOS users to first install brew and tesseract. It may not seem much effort to developers, but these extra steps could detract the user experience and make it less accessible.

On Windows, it is easy to compile the entire Tesseract into one exe file and all I did was copy a compiled Tesseract.exe from another github repo (Probably not the safest idea).

On Mac, I have tried rebuilding all the separate dependencies and Tesseract but to no avail. I was using Python so I couldn't simply import Tesseract dependencies to Python like you would in Swift or Objective C, and the tesserocr interface required interacting with Cython and compiling paths at build time...which would probably not work with relative paths.

otool & install_name_tool

Finally I stumbled upon install_name_tool. Apparently in Unix you could change the dynamic shared libraries at run time and I only needed to use the command install_name_tool to point the dynamic libraries to a relative path after copying the files I needed to run Tesseract.

$ otool -L /usr/local/Cellar/tesseract/4.1.1/bin/tesseract
/usr/local/Cellar/tesseract/4.1.1/bin/tesseract:
    /usr/local/Cellar/tesseract/4.1.1/lib/libtesseract.4.dylib (compatibility version 5.0.0, current version 5.1.0)
    /usr/local/opt/leptonica/lib/liblept.5.dylib (compatibility version 6.0.0, current version 6.3.0)
    /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.7.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

I was not even sure which dylib needed to be changed. That was why I first copied the Tesseract folder and its dependency folders from the installed Cellar folder to a new folder on my desktop.

tesseract
leptonica
giflib
jpeg
libpng
libtiff
openjpeg
webp

Then I ran the cloned tesseract, read the error log, and updated the dylib images that were missing.

$ cd tesseract_standalone/tesseract/4.1.1/bin/
$ sudo install_name_tool -change /usr/local/opt/leptonica/lib/liblept.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib tesseract

Executable Path

Just when I thought I had fixed all the errors and Tesseract was able to run fine, I got a missing image error when I double clicked the tesseract executable and when I called it through pytesseract.

With a bit of searching, I found out that the system couldn't find this library with just the usual relative path notation like ../ but required the use of @executable_path.

sudo install_name_tool -change /usr/local/Cellar/tesseract/4.1.1/lib/libtesseract.4.dylib @executable_path/../lib/libtesseract.4.dylib tesseract

With this extra dylib updated, I can now also run the tesseract executable by clicking on it or through pytesseract.

Here is the final script!

#!/bin/sh

if ! command -v tesseract &> /dev/null
then
    brew install tesseract
fi

if [ "$EUID" -ne 0 ]
  then echo "Please run as root"
  exit
fi

mkdir tesseract_standalone
cp -r /usr/local/Cellar/tesseract tesseract_standalone/
cp -r /usr/local/Cellar/leptonica tesseract_standalone/
cp -r /usr/local/Cellar/libpng tesseract_standalone/
cp -r /usr/local/Cellar/libtiff tesseract_standalone/
cp -r /usr/local/Cellar/webp tesseract_standalone/
cp -r /usr/local/Cellar/openjpeg tesseract_standalone/
cp -r /usr/local/Cellar/jpeg tesseract_standalone/
cp -r /usr/local/Cellar/giflib tesseract_standalone/

cd tesseract_standalone/tesseract/4.1.1/bin/
install_name_tool -change /usr/local/opt/leptonica/lib/liblept.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib tesseract 
install_name_tool -change /usr/local/Cellar/tesseract/4.1.1/lib/libtesseract.4.dylib @executable_path/../lib/libtesseract.4.dylib tesseract
install_name_tool -change /usr/local/opt/leptonica/lib/liblept.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib ../../../tesseract/4.1.1/lib/libtesseract.4.dylib
install_name_tool -change /usr/local/opt/libpng/lib/libpng16.16.dylib ../../../libpng/1.6.37/lib/libpng16.16.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/jpeg/lib/libjpeg.9.dylib ../../../jpeg/9d/lib/libjpeg.9.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/giflib/lib/libgif.7.dylib ../../../giflib/5.2.1/lib/libgif.7.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/libtiff/lib/libtiff.5.dylib ../../../libtiff/4.2.0/lib/libtiff.5.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/webp/lib/libwebp.7.dylib ../../../webp/1.2.0/lib/libwebp.7.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/openjpeg/lib/libopenjp2.7.dylib ../../../openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib ../../../leptonica/1.80.0/lib/liblept.5.dylib
install_name_tool -change /usr/local/opt/jpeg/lib/libjpeg.9.dylib ../../../jpeg/9d/lib/libjpeg.9.dylib ../../../libtiff/4.2.0/lib/libtiff.5.dylib

DEV Community: Mathew Chan

Scheduled Vertical Scaling of AWS Aurora with EventBridge & Lambda in CloudFormation

Aurora under heavy write workloads

Problem: handling periodical heavy writes

Note on Aurora Multi-Master and Aurora Serverless v2

Scheduled Scale-up for Aurora

Scale up - Launching a new instance to the Aurora cluster

Scale up - Failover to newly created instance

Scheduled Scale-down for Aurora

Scale down by failing over to smaller instance

CloudFormation

Remove created RDS in response to failover

CloudFormation permissions and roles

Concerns

1. Downtime

2. Excess replication lag

3. Data replication

Notes on automatic scale up

References

Resolving next-auth "redirect" error when used with Supabase

Python Text Parsing Project: Furigana Inserter for Anki

Furigana

Anki's Furigana Format

Sudachi for Text Segmentation

Sudachi Segmentation Modes

Skip Punctuation and Special Characters

User-defined Mapping and Parsing

Hiragana in the middle of the word

Anki Format

Putting it together

Extra

Anki

Web

Budget Cloud Resources for Hobbyists

Background

Python Anywhere (PA)

Workflow

Digital Ocean (DO)

Workflow

Digital Ocean Spaces

Netlify

Git-based Workflow

Vercel

Git-based Workflow

Railway

Adding self-hosted analytics to your website for free with Umami

Mathew Chan ・ Aug 5 '21

What I have stopped using

MongoDB Atlas

Ending Thoughts

Adding self-hosted analytics to your website for free with Umami

Background

Start an Umami Project on Railway

Login

Add a website

Next JS app

No Social Media trackers!

Migrating your Next.js app from Vercel to Netlify

Background

1. New site from git

2. Essential Next.js Plugin

3. Redirects and rewrites

4. Asset Optimization

Conclusion

Further Reading

Article No Longer Available

Quick way to serve static files with Netlify (100GB free)

Free Options

Paid options

Netlify & Vercel

Netlify Large Media

Netlify sync with Publii

3 Projects I Plan to Release This Year

Introduction

What I have done so far

The State of Language Learning

Project I: Yuzu

Project II: Word List to Ank (web app)

Project III: Twitch Extension for Streamers Who Teach Japanese

Making a Japanese Dictionary Lookup Tool with Sudachi in Python