DEV Community: Dawei Ma

SQLite Renaissance

Dawei Ma — Sat, 25 Feb 2023 14:54:57 +0000

The Story of SQLite
Architecture of SQLite
The Renaissance of SQLite
- Serverless / Edge Computing
- Browser-compatible
- Client/Server
- OLAP
- Distributed
Summary

The Story of SQLite

When I first encountered SQLite, I was struck by its impressive test code, which spans over 150,000 lines of source code and over 90 million lines of test code and scripts. The author, Dwayne Richard Hipp, is a perfectionist who not only developed the underlying storage engine and Parser, but also created the source hosting tool Fossil, as well as the libraries and tools that SQLite depends on, mostly from scratch.

Richard even built the web server, Althttpd, that runs the official SQLite website, with all the code in a single C file and no dependencies on any code libraries except the standard C library. The SQLite website's database is also powered by SQLite, and its dynamic data is rendered in just 0.01 seconds after over 200 SQL statements are queried.

It's remarkable that this development model has been successful, considering the vast majority of the code was written by Richard alone. Although the code is open-source, there is no contribution from the open-source community. In fact, someone has even forked an open-source collaborative version of SQLite, specifically for this purpose: libSQL.

To learn more about the history of SQLite, check out this podcast: The Untold Story of SQLite.

Architecture of SQLite

SQLite is a unique database software that functions differently from most other database systems, such as MySQL, SQL Server, PostgreSQL, or Oracle, which have a client/server architecture. In the client/server model, the client communicates with the database server via a specific protocol, and the server receives and processes the client's requests before returning the results.

Unlike these database systems, SQLite communicates with the application via an in-process approach, functioning as a library. Additionally, the database in SQLite is a single file stored on disk, giving it another advantage over other databases. SQLite is notably fast, especially when executing small SQL query statements. This speed has made it possible for SQLite's official website to obtain dynamic data by querying over 200 SQL statements, without the N+1 query performance overhead of network communication present in other databases.

The architecture of SQLite is illustrated below:

The architecture of SQLite is composed of three main parts: the compiler, the virtual machine, and the storage engine. When an application initiates a query request, the SQL statement is parsed by the compiler, and bytecode is generated. Finally, the virtual machine executes the bytecode, calling the storage engine's interface to read or write data.

The compiler consists of Parser and Code Generator, responsible for parsing SQL statements into abstract syntax trees (ASTs) and converting ASTs into bytecode, respectively. Code Generator also generates query plans.

The Virtual Machine is a Register-Based VM responsible for executing bytecode, and it also handles query optimization. Readers interested in this part can read this article: How the SQLite Virtual Machine Works.

The storage engine's main responsibility is reading or writing data. It includes the B-tree, Pager, and OS Interface (also called VFS).

B-tree: SQLite's indexes are stored in B-tree data structure and table data are stored in B+tree data structure. For readers interested in this topic, I recommend reading this article: SQLite Internals: Pages & B-trees.
Pager: Serves as an abstraction layer between the B-tree module and VFS module. It provides the function of reading, writing, and caching disk pages, achieving atomicity, isolation, and persistence in SQLite. Pager offers two concurrent access modes: Rollback Journal and Write-ahead Log. The Write-ahead Logging mode provides better scalability and allows reading data concurrently while writing data. Although only one write thread is allowed to update the write-ahead log file at a time, the configuration of busy_timeout allows multiple write threads, but execution remains serialized. For more information, refer to these articles:
- How SQLite helps you do ACID
- How SQLite Scales Read Concurrency
OS Interface (VFS): To provide portability across operating systems, SQLite uses an abstraction layer called VFS. VFS provides methods for opening, reading, writing, and closing disk files, as well as other operating system-specific functionality. For readers interested in this topic, I recommend reading the SQLite documentation on VFS.

Above is a brief introduction to the SQLite architecture. If you want to further understand the internal implementation details, you can read this open-source e-book: SQLite Internals: How The World's Most Used Database Works.

This understanding will help you appreciate the following open-source projects that creatively utilize SQLite, showcasing its versatility.

The Renaissance of SQLite

SQLite, a software that is over 20 years old, is often seen as a database for simple local storage or testing, and rarely used in production. However, there are several interesting projects that have revived SQLite and have been hotly discussed in forums such as Hacker News. Let's take a look at some of these projects.

Serverless / Edge Computing

Jamstack architecture for serverless applications involves publishing static pages to a CDN and using APIs to provide dynamic updates. This architecture can greatly enhance scalability for business systems. However, there are limitations to this approach. Data must be stored in a separate hosted database, which can be expensive, and the database can become a performance bottleneck. This is particularly true when business systems are deployed to multiple regions. But what if the business system instance and the database were running on the same server? This approach has the potential to eliminate the network overhead of a single-node database, offering a more efficient solution.

SQLite is a Serverless database, which runs in the same process as the application. Compared to other databases, SQLite is faster than those with C/S architecture for network communication. However, a common problem when multiple instances are reading and writing to the same database needs to be addressed. One possible solution is Litefs, a project developed by the author of Litestream after joining Fly.io.

Litestream is capable of controlling the management of wal log files by taking over the SQLite checkpointing process. When operating in Write-ahead Log mode, Litestream can continuously copy wal log files to the backup location, such as S3, enabling online streaming backup of SQLite database files. For more information, see this document detailing how it works.

Litefs takes it a step further than Litestream by providing a FUSE-based file system to SQLite as the VFS layer. Litefs can replicate the collection of pages related to a transaction at the page level and complete the cross-node synchronization of data by packaging these page collections into a file package in LTX data format and sending this package to the read-only node via HTTP protocol.

In the distributed cluster of Litefs, only the master node can write data, and the read-only node can let the client write data only at the master node by forwarding the address of the master node to the client. The master node is elected by obtaining distributed leases (Distributed leases) from Consul to reach consensus, and static master nodes can also be set.

For details about Litefs' specific architecture, please refer to this article. A deployment case can be found in this thread, which discusses a migration from a Postgres cluster to distributed SQLite with Litefs.

Cloudflare has also released a similar commercial solution, Cloudflare D1.

Browser-compatible

SQLite can be run in the browser using WebAssembly (WASM) technology, and there are two projects that enable the use of SQL within the browser.

sql.js allows JavaScript to download the SQLite database file into the browser's memory via network request. SQL can then be used to retrieve data results from the SQLite database file all within the browser.
absurd-sql is similar to sql.js, but it can use the browser's IndexedDB as persistent storage and can read and write SQLite database files.

What are the advantages of using SQLite in the browser? Take this open source project of mine Invest Alchemy as an example. It is an ETF portfolio management system that needs to manage multiple ETF portfolios, and all the data of each ETF portfolio is stored in a SQLite database. The location of this database file is stored in AWS S3. Every day, a scheduled program automatically downloads all SQLite databases to the AWS S3 bucket. After which, the data of these portfolios is updated and finally uploaded to S3.

When a user views a page of a portfolio, such as this portfolio, the page will first download the SQLite database from S3 to the browser's memory when it is first rendered. Later, it uses sql.js to initialize the SQLite database and launches multiple SQL queries to get the data results, then renders the page.

The advantage of this architecture is that the browser page only needs to make one query request to get all the data for the entire portfolio. When using a traditional database, this is a matter of cost, and each query must be transmitted over the network, which increases the load time of the page.

Finally, SQLite's single database file approach brings good isolation. For example, in Invest Alchemy, one database represents one portfolio. It is possible to store all the personal data of one user in one database and then store these databases in different directories in AWS S3 so that the data of different users can be well isolated.

Client/Server

SQLite can be used as a Client/Server architecture database, although this usage can come at the cost of losing some of the advantages of SQLite and increasing the network overhead. Nevertheless, it can be useful in certain scenarios, such as a read-only data source or as a data cache where the overhead is minimal.

Here are a few projects that demonstrate the use of SQLite in a client-server architecture:

Postlite: A web proxy library that supports PostgreSQL's communication protocol and uses SQLite as storage on the backend.
sqlite3vfshttp: A SQLite VFS that supports accessing SQLite database files via HTTP protocol. Compared to sql.js, which needs to download the whole SQLite database file, this library only needs to specify the HTTP range header by the client to get the data of the specified range. In a large database file, this optimization can save a lot of network overhead.
Cloud Backed SQLite: Officially supported cloud SQLite, supports Azure and GCP, can read or write directly through Storage Client database without downloading the whole database.

OLAP

SQLite is commonly used as an OLTP database but is rarely used for OLAP since its table data storage is row-based rather than column-based. To fill this gap, DuckDB was developed, which has a similar architecture to SQLite but uses columnar storage, making it perfect for OLAP scenarios.

However, starting from version 3.38.0, SQLite has improved the performance of large analytic query statements using Bloom Filters, which is also designed to support OLAP scenarios. More information on this optimization and a comparison with DuckDB can be found in this paper.

Distributed

The in-process architecture of SQLite is not relevant to distributed databases, which are a complex field. Distributed databases offer elastic expansion and high availability, but due to the complexity of distributed transactions, conventional practice involves vertical scaling of a single machine as much as possible. Horizontal scaling (slicing) is not a feasible solution. Nevertheless, there are some impressive projects that have brought SQLite into the realm of distributed systems.

rqlite is a distributed database that uses Raft to solve the problem of achieving consensus among nodes in a cluster. In this architecture, writes are performed through the Leader node, and the other replica nodes can pass write requests to the Leader node. Meanwhile, reads can be done by any node, making this a Leader-Replica style of distributed architecture.

However, because writes are still performed by the Leader node and there is additional overhead due to consistency checking, rqlite does not significantly improve write throughput compared to the In-Process standalone SQLite.

The rqlite data API is publicly available via HTTP, which makes it a Client/Server style of architecture. To achieve inter-node synchronization, rqlite copies commands between nodes. For example, when a write command is sent to the Leader node and submitted to Raft Log, the Leader node copies the write command to other nodes.

If you're interested in learning more about rqlite's design, you can check out the documentation available here: Rqlite Design.

mvsqlite: The subtlety of this project lies in its use of FoundationDB as the VFS layer of SQLite. It makes great use of the distributed features provided by FoundationDB, such as optimistic lock-free concurrency, distributed transactions, synchronous asynchronous replication, and backup and recovery, to achieve a distributed SQLite with MVCC support. This includes not only Leader-Replica, but also parallelization of multi-node writes, which increases the throughput of writes and enables time travel at the database level (Time travel) thanks to its implementation of MVCC.

To learn more about the implementation details of mvsqlite, we recommend reading these two articles by the author:

Compared with Litefs, which is also distributed through the VFS module, mvsqlite requires additional deployment of FoundationDB cluster and mvstore stateless instances, resulting in higher deployment and O&M costs.
Interestingly, FoundationDB used SQLite as the SSD storage engine before 7.0.0, but after that, FoundationDB implemented its own storage engine called Redwood.

cr-sqlite: To achieve ultimate consistency, the previous project, rqlite, used a consensus algorithm to select a particular leader. However, what if multiple writers need to make changes to the same database at the same time without conflict? Fortunately, a data structure called CRDT (Conflict-free Replicated Data Type) exists in the real-time collaboration field to solve this problem. The cr-sqlite project has ingeniously integrated CRDT into SQLite through its runtime extension, resulting in the same cluster multi-node concurrent writing feature as mvsqlite. For more information, I recommend reading the author's article on Why SQLite? Why Now?.
Bedrock: Bedrock is an advanced web and distributed transaction layer built with the help of SQLite. It functions as a distributed relational database system that has been designed primarily for replicating offsite data. Using a peer-to-peer style of distributed architecture, information is eventually stored on a private blockchain across all nodes. 🤯

The Paxos consensus algorithm is used by Bedrock in order to select Leaders from clusters. They are responsible for organizing two-phase commit transactions across different systems. Moreover, its synchronization engine is based on the implementation of blockchain technology.

Each node contains an internal table referred to as the journal – it consists of 3 columns: id, query, and hash. Every time a query is committed into the database, a new row is added to the journal. This records the query and computes a unique hash value according to the most recent row.

When a server links up with a cluster, the newest hashes and IDs are shared. If two distinct servers dispute over the assigned hash corresponding to the ID, they will recognize they were “forked” at some point and stop exchanging messages. In such situations, the leader decides which fork is deserving of becoming the master branch.

Summary

It is no secret that SQLite has become a highly innovative project in many areas. This is in large part due to the high-quality of its source code, boasting almost 100 million lines dedicated to testing and verifying its robustness. This minimalistic architecture constrains the total source code to just over 150,000 lines, which enables individual developers to individually innovate on it more easily compared to databases like MySQL (with over 4 million lines), Oracle (with over 10 million lines), or Postgres (over 1 million lines).

This simple layout also makes SQLite a great platform for experimentation; developers can implement a vast array of modifications with relative ease. As edge computation and serverless deployments run off of Content Delivery Networks, SQLite - being a lightweight relational database - will only see further use cases and innovation flow into it. Moreover, since its codebase is straightforward, those wishing to learn about SQLite principles can do so quickly.

Adventures in K8S Cloud Native App Development

Dawei Ma — Thu, 11 Nov 2021 15:47:18 +0000

Note: This post is a pairing between the author and GitHub Copilot. Copilot did about 5% of the work on this post. The author also partially documented Copilot's work in this Tweet.

As an amateur indie developer, I don't invest much time or money in development, so I have two very basic requirements for my side project tech stack:

The tech stack can greatly improve my development efficiency.
The tech stack does not require a lot of money.

The first point is that I would choose to use a more efficient tech stack, including programming language, ecology, architecture, etc.

I tried Elixir because it is based on Erlang platform, which has a powerful concurrency model and expressive language syntax, and can take advantage of Erlang/OTP ecology, which allows me to develop a product online more efficiently.
I tried Serverless because many personal products have few initial users, and the traditional way of renting VPS would waste a lot of resources, and the elastic scaling ability and availability of Serverless are more power than VPS, so I will use Serverless to develop some small products.
I tried IaC because it can build infrastructure in a declarative way and also manage infrastructure versions so that my investment in infrastructure is one-time and I don't have to repeat manual operations to provision the server environment each time.

The second impact on me is that I will avoid using large, resource-intensive tech stack and instead look for cheap, lightweight alternatives. This also means that I won't be looking for high SLA, cost effectiveness is my main goal. Choosing an industrial-grade tech stack, sacrificing some availability, allows me to accept the cost on the one hand, and get industrial-grade scalability on the other. This is reflected in my cost analysis of Serverless, where I choose the service components that are affordable and the most cost-effective billing service.

My initial impression of the K8S is that it doesn't meet my requirements on either of these points. That's why I never tried it on my side project. Until I came across this long post The Architecture Behind A One-Person Tech Startup.

The author describes his experience using K8S on his personal projects, which is resource intensive but brings great scalability and reliability.

Architecture Evolution History

Application architectures, in terms of composition pattern, are divided into monolithic and distributed architectures.

Primitive distributed architectures predate even monoliths, as early computers were all poor in performance and could not meet the expanding human demand for computing power, which in turn led to the evolution of application architectures. As the performance of single computers improved, and the original distributed technologies were so complex that monolithic architectures became popular for a long time until the performance of single computers could not meet the massive amount of information needed to compute the explosive growth of human society.
Distributed architectures provide higher performance, higher availability, and higher scalability by coordinating the use of the computing power of multiple computers. However, due to its complexity, the evolution of distributed architectures is further divided into these phases.
- Phase 1: Service-Oriented Architecture (SOA). Service-oriented architecture is an architectural model that successfully addresses the main problems of distributed services in a concrete and systematic way at a time. However, this architecture requires a lot of time and effort from application developers to learn the framework itself, and the architectural design of this architecture pattern is more complex and too costly to promote.
- Phase 2: Microservice Architecture. Microservices is an architectural style of building a single application through a combination of multiple small services that are built around business capabilities rather than specific technical standards. This solves the complexity problem of SOA and allows business developers to focus more on business development. But the problem with microservices is that business developers still have to deal with these problems that need to be solved for distributed architectures such as service discovery, error tracking, load balancing, transport communications, etc.
- Phase 3: Cloud Native Architecture. Cloud-native architecture is the development from the software level alone to cope with the problems that cannot be solved by microservices architecture to the soft and hard integration (software-defined computing, software-defined networking, and software-defined storage), and the combined efforts to cope with the general problems of distributed architecture. Using technologies such as containers, virtualization technology, immutable infrastructure, service mesh, declarative APIs, K8S provides out-of-the-box elastic scaling, service discovery, configuration center, service gateway, load balancing, service security, monitoring and alerting, fault-tolerant processing and other functions. These technologies enable the construction of loosely coupled systems that are fault-tolerant, easy to manage and easy to observe. Combined with reliable automation, cloud native technologies enable engineers to easily make frequent and predictable major changes to the system.

Cloud Native Era

As the virtualized infrastructure expands from containers of individual services to clusters of services, communication networks and storage facilities consisting of multiple containers, the line between software and hardware is blurred. Once virtualized hardware is able to keep up with the flexibility of software, technical issues that are not business-related can be stripped away from the software and silently resolved within the hardware infrastructure, allowing software to focus solely on the business and truly "build teams and products around business capabilities. (From Phoenix Architecture/Post-Microservices Era)

Kubernetes(K8S)

K8S Design

K8S creates a DSL language in which users define every resource (e.g., compute, network, storage, routing, secrets, certificates) used in a distributed system architecture declaratively. When the user defines the state of the resources they expect, K8S automatically helps them create those resources and manages them automatically.

Resources is an extremely common term in Kubernetes. Broadly speaking, all aspects of the Kubernetes system that you can touch are abstracted into resources, such as resources for workloads (Pod, ReplicaSet, Service, ......), resources for storage (Volume, PersistentVolume, Secret, ......), resources for policies (SecurityContext, ResourceQuota, LimitRange, ...), and resources for policies (SecurityContext, ResourceQuota, LimitRange, ...). resources (Volume, PersistentVolume, Secret, ......), resources representing policies (SecurityContext, ResourceQuota, LimitRange, ...), and resources representing identities. ...), resources representing identities (ServiceAccount, Role, ClusterRole, ......), and so on. The "everything is a resource" design is an essential prerequisite for Kubernetes to be able to implement declarative APIs, and Kubernetes uses resources as a vehicle to build a domain-specific language that encompasses both abstract elements (e.g., policies, dependencies, permissions) and physical elements (e.g., software, hardware, networks). The collection of descriptions of resource states, from the state of the entire cluster or even a cluster federation down to a memory area or a small number of processor cores, through the relationship of resource usage between different layers, together form a panoramic picture of the working operation of an information system. (From Phoenix Architecture/Unchangeable Infrastructure)

Benefits

Ability to build fault-tolerant, easy-to-observed applications
Enables applications to be managed in a unified manner
Enables applications to scale resiliently
One-click application installation and deployment (Helm)

Disadvantages

High resource costs. Both the Master and Worker nodes of K8S require a certain amount of compute resources.
K8S redefines many abstract technical concepts and has a high learning curve.

Cost analysis of K8S hosted on cloud platforms

My choice of Kubernetes hosting solution for different cloud platforms is mainly based on cost considerations. This K8S Cluster Cost Compare document provides a cost analysis of different cloud platforms (AWS/Azure/GCP/ DigitalOcean/Vultr) for Kubernetes hosting solutions.

I ended up with the cheapest DigitalOcean cloud platform, where Master Control Plane Basic is free, Worker nodes are 2-core 4GB RAM machines in Singapore region ($20/month), and a $10/month Load Balancer fee. The total cost for a month is $30/month.

Since the Worker node needs to install some K8S services such as kube-proxy, core-dns, etc., there are 12 pods, which take up half of the memory of the Worker node. This leaves 2GB of resources left for user application usage.

Application Architecture

The above diagram shows the development and deployment process of this cloud-native application and the architecture of each internal service deployed by K8S.

Development and deployment process. The push of the code to GitHub triggers two actions.
- Vercel detects changes to the front-end code and automatically deploys it to Vercel's CDN if there are any changes.
- GitHub Actions detects changes to the back-end code, builds the image and publishes it to GitHub Packages, then automatically creates a new K8S Deployment and redeploys the back-end service.
Request flow processing. When a user visits a website, DNS is parsed by Cloudflare and the browser sends two requests to:
- Vercel side: the browser pulls static page resources.
- K8S side: the request is forwarded to the Service of ExternalName type of default Namespace after the Ingress rule is parsed by Load Balance of K8S, and then forwarded to the Service of the back-end service (Namespace is free4chat), and finally forwarded by the Service to the Container of one of the Pods. The Container is our backend business application.

Implementation

The final result is available at: Online version.

The source code is available at: code repository.

Pre-requisites

Pre-requisite knowledge

If you don't know anything about K8S, you can start by watching this high-quality introductory video: Kubernetes Tutorial for Beginners [FULL COURSE in 4 Hours]. Make sure you understand the basic K8S concepts: Namespace, Deployment, Service, Pod, Node, Ingress before you get into the real thing.

Pre-requisites

You will need to register for the following account first.

DigitalOcean
Cloudflare
A domain name.

And install these software.

kubectl
doctl
helm

Project directory structure

.
├── .github
│   └── workflows
|       └── workflow.yml
├── Makefile
├── backend
├── frontend
└── infra
    ├── Dockerfile.backend
    ├── k8s
    │   ├── free4chat-svc.yaml
    │   ├── ingress-free4-chat.yaml
    │   ├── ingress_nginx_svc.yaml
    │   └── production_issuer.yaml
    └── tools
        └── nsenter-node.sh

The overall project is divided into frontend, backend, and infra parts, and this article focuses on the infra part, which is the K8S deployment. infra does not use IaC because K8S itself is a declarative yaml files, and there is no need to add complexity to use IaC if you are not using some cloud hosting service. the CI/CD part is done by using GitHub Actions.

Dockerfile

The backend service is a Golang application, and the packaged Dockerfile is here. I also made a simple configuration of Makefile for compiling the backend service. Local deployment of backend services using Docker can use this Makefile.

Configuring K8S

Creating a K8S Cluster

Creating a K8S Cluster at DigitalOcean is a very simple task, you just need to select the region (depending on where your business users are located) and the specifications of the Worker Node (depending on your cost budget) to create a Cluster.

Connecting to a K8S Cluster

Configure K8S using doctl:

doctl kubernetes cluster kubeconfig save use_your_cluster_name

Namespace

Namespace is a mechanism used by K8S to isolate groups of resources within a single cluster. For example, we can create different business Namespace in the same cluster, and under this Namespace, we can store Pods, Services, Deployment and other resources related to this business, if we want to delete the resources related to this business, we just need to delete this Namespace.

By default, K8S has a kube-system Namespace, which holds K8S-related resources. There is also a default Namespace, which holds the resources created by default (without the Namespace).

Backend Service

First create a Namespace for the backend service.

kubectl create namespace free4chat

Then create a backend Service template free4chat-svc.yaml.

apiVersion: v1
kind: Service
metadata:
  name: apifree4chat
spec:
  ports:
  - port: 80
    targetPort: 8888
  selector:
    app: apifree4chat
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apifree4chat
spec:
  selector:
    matchLabels:
      app: apifree4chat
  replicas: 1
  template:
    metadata:
      labels:
        app: apifree4chat
    spec:
      containers:
      - name: echo
        image: <IMAGE>
        resources:
          requests:
            memory: "128Mi"
            cpu: "250m"
          limits:
            memory: "500Mi"
            cpu: "1000m"
        ports:
        - containerPort: 8888

The reason this is a template is that there is a <IMAGE> placeholder in the image section that will be replaced with the real image when GitHub Actions are deployed later.

This template defines a Deployment and Service resource; Deployment defines the CPU and memory limits for Pod instances, the number of instances, port mapping, and container images, etc. Service defines the domain name and port for accessing backend services inside the Cluster.

Finally, GitHub Actions deploys this Service and Deployment to the free4chat namespace in the K8S Cluster.

Ingress Controller

How do you get external traffic to the backend service when you have a Service for the backend service? This is what K8S Ingress does. Our first choice is to install the Ingress Controller, which comes in many types, such as HAProxy, Nginx, Traefik, etc. We choose Nginx here. Find Nginx Ingress Controller on the DigitalOcean K8S administration interface, and click Install.

This will automatically create an ingress-nginx namespace and a DigitalOcean Load Balance service, which costs $10/month and has a separate IP address (available in the DigitalOcean administration interface). We will use this IP in our DNS configuration later.

Now we need to create an ingress rule under default Namespace to forward the LoadBalance traffic to the backend service, the configuration file is ingress-free-4-chat.yaml.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-free4chat-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod # This is the ClusterIssuer for cert-manager, which is used to automatically generate SSL certificates
spec:
  tls:
  - hosts:
    - api.k.free4.chat
    secretName: api-free4chat-tls
  rules:
  - host: api.k.free4.chat # Back-end service domain name
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: apifree4chat
            port:
              number: 80
---
kind: Service
apiVersion: v1
metadata:
  name: apifree4chat
spec:
  type: ExternalName # Because the backend service is not in the default Namespace, it needs to be forwarded to the backend service in the apifree4chat Namespace through the ExternalName service.
  externalName: apifree4chat.free4chat.svc.cluster.local # Back-end service domains across Namespace

This configuration file will generate two resources, an ingress rule and a Service of type ExternalName. we will use kubectl to create this resource after configuring the ClusterIssuer for cert-manager.

Cert Manager (HTTPS)

The SSL certificate for the domain is automatically generated and updated in K8S via Cert Manager, where we use the Let's Encrypt service to issue the certificate for us.

First, install the Cert Manager application via Helm with one click.

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.2.0 --set installCRDs=true

After executing these commands you need to create a ClusterIssuer resource that issues SSL certificates for the production environment, the configuration file is production_issuer.yaml.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # Email address used for ACME registration
    email: your@email.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Name of a secret used to store the ACME account private key
      name: letsencrypt-prod-private-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

In DigitalOcean, in order for Cert Manager to be self-check, Pod-Pod communication must be enabled through the Nginx Ingress Controller for Cert Manager to work properly. To create a Service resource for the K8S approach certificate, the configuration file is ingress_nginx_svc.yaml.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: 'true'
    service.beta.kubernetes.io/do-loadbalancer-hostname: "k.free4.chat"
  labels:
    helm.sh/chart: ingress-nginx-2.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.34.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller

Creating resources

After the above steps we have some declarative K8S resource creation profiles, now it's time to actually start creating these resources:

kubectl apply -f production_issuer.yaml # Create the ClusterIssuer resource for issuing SSL certificates
kubectl apply -f ingress_nginx_svc.yaml # Create a Service resource that resolves Pod-Pod communication
kubectl apply -f ingress-free-4-chat.yaml # Create ingress rule resource

When these commands are executed, we are still missing some steps.

The creation of the backend service resource. We'll create this via GitHub Actions.
DNS domain configuration. We will configure this on Cloudflare.

DNS configuration

Configuring DNS resolution on Cloudflare.

The reason for this is that we cannot create two different IP records for a primary domain at the same time, so we have to give the back-end domain name to solve this problem.

Finally, we create two CNAME records, which are

api -> api.k.free4.chat: the domain name of our backend service API.
www -> www.free4.chat: our main domain name.

At this point we can access https://www.free4.chat. But https://api.k.free4.chat doesn't work yet, because the backend service hasn't been created yet. So next we need to create the backend service via GitHub Actions.

GitHub Workflow

The benefit of creating backend services via GitHub Actions is that it automates deployments and automatically triggers GitHub Actions to build new images and create new backend services when changes are made to the backend code.

To create a GitHub Workflow, just create .github/workflows/workflow.yaml.

name: DO_K8S_Deploy

on:
  push:
    branches:
      - main
    paths:
      - 'backend/src/**'
      - 'infra/Dockerfile.backend'
      - '.github/workflows/**'

# A workflow run is made up of one or more jobs that can run sequentially or in parallel.
jobs:
  # This workflow contains a single job called "build".
  build:
    # The type of runner that the job will run on.
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:

    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it.
    - name: Checkout master
      uses: actions/checkout@main

    # Install doctl.
    - name: Install doctl
      uses: digitalocean/action-doctl@v2
      with:
        token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}

    # Build a Docker image of your application in your registry and tag the image with the $GITHUB_SHA.
    - name: Build container image
      run: docker build -t ghcr.io/madawei2699/apifree4chat:$(echo $GITHUB_SHA | head -c7) -f ./infra/Dockerfile.backend .

    - name: Log in to GitHub Packages
      run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin

    - name: Push image to GitHub Packages
      run: docker push ghcr.io/madawei2699/apifree4chat:$(echo $GITHUB_SHA | head -c7)

    - name: Update deployment file
      run: TAG=$(echo $GITHUB_SHA | head -c7) && sed -i 's|<IMAGE>|ghcr.io/madawei2699/apifree4chat:'${TAG}'|' $GITHUB_WORKSPACE/infra/k8s/free4chat-svc.yaml

    - name: Save DigitalOcean kubeconfig with short-lived credentials
      run: doctl kubernetes cluster kubeconfig save --expiry-seconds 600 ${{ secrets.CLUSTER_NAME }}

    - name: Deploy to DigitalOcean Kubernetes
      run: kubectl apply -f $GITHUB_WORKSPACE/infra/k8s/free4chat-svc.yaml -n free4chat

    - name: Verify deployment
      run: kubectl rollout status deployment/apifree4chat -n free4chat

The only thing you need to do here is create the CLUSTER_NAME and DIGITALOCEAN_ACCESS_TOKEN environment variables in the Actions secrets for this repo in advance for GitHub Actions to use. where DIGITALOCEAN_ACCESS_TOKEN is the DigitalOcean API Token and CLUSTER_NAME is the name of our Kubernetes Cluster on DigitalOcean.

This way, whenever a code update is pushed to GitHub, a new service (including front-end and back-end) is automatically built and published to Vercel and DigitalOcean K8S.

When you are here, our application goes live!

One More Thing

Logging: Traditional ELK requires a lot of server resources and is not suitable for a lightweight cluster like ours. The easiest way is to run multiple Pods to see the logs, and there is a stern tool that helps us to query the logs across multiple Pods.
Monitoring and alerting: We can monitor our services by installing Prometheus and Grafana, and we can send alerts through Prometheus' Alert Manager. However, if the entire cluster is down, the monitoring and alerting service installed into the cluster will not work, so the best practice is to use an external monitoring and alerting service. This can be done using New Relic or a similar service.
Error tracking: Integrating Sentry will enable backend service error tracking.

Summary

At this point we have built a K8S cluster from scratch (without including the K8S Master control plane). Let's think about the question, what problem does K8S help us solve?

Let's start by thinking about the 12 Factor that are often considered in modern software development.

Codebase
Dependencies
Config
Backing services
Build, release, run
Processes
Port binding
Concurrency
Disposability
Dev/prod parity
Logs
Admin processes

K8S offers solutions to all of these factors, directly or indirectly, and the ecosystem around it allows engineers to build robust software that meets these software design factors at low cost.

I think this is the reason why K8S is called the operating system on the cloud.

References

Cloud IDE

Dawei Ma — Tue, 24 Aug 2021 11:32:42 +0000

A while back, the GitHub Twitter posted the following Tweet.

If you press . on any GitHub Repo page, you will automatically be redirected to the github.dev which is a web version of VSCode and will automatically clone the code of the repo. In this web VSCode you can even install some specific plugins (you can't install plugins that require external dependencies) to make it easier to read the code. Because this site is official, VSCode is automatically bound to your GitHub account, so developers can read, edit, and commit code in it without having to open a local IDE. github1s), an open source project with similar functionality.

VSCode's team leader Erich Gamma (one of the authors of JUnit, one of the authors of Design Patterns, and an Eclipse architect) joined Microsoft in 2011 with the following job description.

Envision new paradigms for online developer tooling that will be as successful as the IDE has been for the desktop.

Then came the birth of VSCode. We can say that VSCode is designed to be a cloud-based IDE from its inception.

Why use a cloud IDE, it is due to some problems in local development environment, such as

Environment differences: There are some differences between Mac and Linux, especially when using popular Linux kernel-based technologies like Docker on Mac, which can make the whole experience worse.
Performance issues: Local development machines generally need to run a lot of software, such as some office and communication app, so the performance is not high enough to stretch.
Stability problem: The local development machine takes a long time to restart, and it takes a lot of time to set up the development environment after each restart, which wastes a lot of time.
Dependency issues: If the development environment relies on some specific cloud infrastructure, network communication can be troublesome. Whereas on cloud hosting, it is naturally in a network environment with other cloud infrastructures and the environment is simple to set up.
Network issues: Modern software development is standing on the shoulders of giants, and a lot of software relies on a large number of libraries, frameworks, and runtime, and these dependencies require fast network speeds to download. Generally cloud hosting has better network performance than a home or office network.
Security issues: There is a risk of leakage of code or keys placed in the local development environment, such as the loss of code or keys when a developer's development machine is stolen.
Storage issues: Local development machines have limited disk storage and do not scale well. And the cloud host's disks are easily expandable.

The ultimate solution to these problems is to move the development environment to the cloud, and the most important thing to develop in the cloud is the need for a good IDE support, which has led to a strong demand for cloud IDEs in the industry.

Cloud IDE

Before we talk about the cloud IDE let's understand some of the main functional points of the IDE, as shown in the following figure.

A good IDE should of course let the programmer write the code to write the cool, see the code to see the smooth. And to achieve this purpose, it is essential to support the following feature points.

Excellent text editing features, supporting keyboard custom layout.
Code hint functions, such as syntax highlighting, code jumping, error hints, etc..
Debugging features.
Multi-programming language support.
Code completion function.
Code refactoring capabilities.
Extension capabilities, support for user-defined or provided plug-ins.
Good ecology.

In desktop IDEs, these features are not a problem, and there are many IDEs that support them, such as Visual Studio, Eclipse, IntelliJ IDEA, NetBeans and Xcode. But in the dimension of online support, none of these older IDEs can.

The requirements for cloud IDEs in the early industry were also not high, so there were about three broad categories of cloud IDEs, as follows.

Online editors. Web online editors, mainly CodePen and JSFiddle, make it easy to do online development of front-end pages. But this is far from the desktop IDE experience.
Repl.it and Jupyter are the main online REPL. REPL is at most one of the many features supported by desktop IDEs, and its usage scenario is suitable for writing some validation type There is still a long way to go before an engineered code development experience.
Cloud IDEs with limited functionality, mainly AWS Cloud9, are already very good for code development, even seamlessly using cloud infrastructure, and are suitable for collaborative code development at scale. However, these cloud IDEs are generally not scalable, for example, some plugins cannot be installed.

For example, Github Codespaces uses VSCode combined with Azure cloud servers to give developers a desktop IDE experience and the ability to install plugins. Also, in a 2021 Developer Survey questionnaire from StackOverflow, the most popular developer community, the Integrated development environment (IDE) section of a StackOverflow 2021 Developer Survey survey (with more than 80,000 developer survey responses), VSCode was voted the most popular IDE with 71% of the votes (up to 50% in 2019).

It seems that VSCode achieves the ultimate goal that cloud IDEs are trying to achieve: the same development experience as desktop IDEs. The question here is why VSCode?

Why VSCode

Good Design

The VSCode remote development model is shown in the following figure.

The CS architecture is designed to give VSCode the ability to use remote servers or containers. The local VSCode is only responsible for the UI interface and theme display, while other things such as plug-ins, program runs, terminal processes and debuggers are run on the remote server. The separation of interface display and computation is a very important point for implementing a cloud IDE.

The CS architecture design is also reflected in the code prompting. By developing the Language Server Protocol standard protocol, the VSCode core does not need to parse ASTs of multiple programming languages or implement multiple programming language parsers, but delegates these functions to the plug-ins of each language, ensuring that the core is very small and stable.

The same design is also reflected in the Debugger and the Debug Adaptor Protocol standard protocol.

More analysis of the architecture can be found in my article VSCode Plugin Development Notes.

Cross Platform

VSCode was formerly known as monaco-editor online editor. As a web software, it need to use Electron technology to achieve cross-platform functionality. So the desktop VSCode and the server-side Web version of VSCode are actually one code base.

Because it is a web software, there is a third-party Repo code-server and a VSCode running in the browser.

Open source

VSCode Without open source, it could have ended very differently. It was Microsoft's gorgeous turnaround and enthusiastic embrace of open source that opened the door for VSCode to go global, or it could have ended up as one of Microsoft's many internal projects.

Build Cloud IDE based on AWS with Pulumi

Thanks to VSCode's open source and web features, we can quickly build a VSCode-based personal cloud IDE that is comparable to Github Codespaces, but much cheaper.

My implementation can be found in this Repo. The architecture is as follows.

Prerequisites.

An AWS account with the AWS CLI installed and AWS Credentials configured locally. The AWS account needs to have rights to access to EC2.
A Pulumi account and create a project; (if you are not familiar with Pulumi, you can refer to this article Implementing static blog access statistics based on Serverless)

Very simple to use (thanks to the power of Pulumi and AWS CLI).

git clone https://github.com/bmpi-dev/code.bmpi.dev.git
cd code.bmpi.dev/server
pulumi up # Set up AWS EC2 with Pulumi
./run work # Open remote VSCode
./run rest # Shut down the remote VSCode

If ./run open_tunnel cannot connect to the tunnel while the server is still up, you can run it again after the server is up to establish the tunnel connection.

You need to enter the VSCode login password for the first time access, by executing sh connect-server.sh and then execute cat ~/.config/code-server/config.yaml | grep password: to get the login password.

You can start using the remote VSCode by accessing http://localhost:8888/ through your browser.

If you don't need this environment for a while, remember to hibernate it via ./run rest to hibernate the cloud server. After the server is shut down, AWS does not bill the EC2 instance, but only the storage volumes for a very cheap fee.

If you don't need the environment at all and want to destroy all resources so that AWS doesn't continue to charge, just run pulumi destroy to delete all AWS resources.

About fees

Take an AWS EC2 T2.Medium instance (2 cores 4GB RAM + 50GB storage) as an example. For 5 hours of development per day and 100 hours per month for 20 days, the total cost is $0.0464 * 100 + $0.1 * 50 = $9.64. The same server configuration with Github Codespaces costs $21.50, 2.23 times bigger than our.

The future of Cloud IDE

Cloud IDE represents the future of an R&D model. The possible development trends of this R&D model are as follows.

Standardized development environment. A set of cloud IDE development environment can be standardized for batch configuration and used out of the box, significantly reducing the time consumption of developers in configuring the development environment.
Customized development environment. The development environment can be customized to meet the needs of different types of projects.
Elastic development environment. The configuration of the development environment relies on the automatic elastic expansion of the cloud service, and the configuration can be dynamically adjusted to meet the dynamic needs of the development environment for resource allocation.
Intelligent development environment. Relying on the cloud server's machine learning analysis of specific code repositories, it can better achieve intelligent tips to assist development, similar to GitHub Copilot.
Secure development environment. The code and infrastructure configuration are stored on the cloud server, which can greatly reduce the risk of code or environment key leakage caused by developer negligence. With a good system security configuration of the cloud server, the security risk of the development environment can be reduced.
Ready-to-use development environment. No need for a specific development machine, just a computer with a browser to access the cloud IDE to start development.

In a nutshell.

Cloud IDE, Coding Anytime Anywhere.

Reference

Internationalization(i18n) and Localization(L10n)

Dawei Ma — Thu, 22 Jul 2021 15:14:06 +0000

internationalization(i18N)
- Problems to be solved for internationalization
- Text Encoding
- locale
- Language and Country Codes
- gettext
- Internationalization Process
Localization(L10N)
- Localization Process
- Develop a localization strategy
- Region and Language
- Add new region/language/service
- Incremental Localization
- Management of translation
- Localized implementation
- Localized multilingual implementation
- The Challenges of Localization
- Do you need to consider SEO
- Localization of product design
- Localization under Microservices
- Localized technical or business standards development
- Development Environment and Business Processes
- Static text processing
- Whether to store language and region settings
- Localization of back-end services
- Localization of third-party services and resources
- Release Process
- Localization under micro front-end architecture
- Localized testing
- Localization Platform

A successful product needs to go global through many stages, from the perspective of software development there are two main processes: internationalization and localization.

A language environment is the use of a specific language or language variant within a country or geographic region, which determines the format and parsing of dates, times, numbers and currencies, as well as the various measurement units and translated names of time zones, languages, countries and regions. Internationalization enables a piece of software to handle multiple language environments, localization enables a piece of software to support a specific regional language environment. This means that the process of globalization is to first make the software internationalized, and then to do the localization implementation so that it can support a specific language environment in a specific region.

They are often abbreviated as i18n (18 means that there are 18 letters between i and n in the word "internationalization") and L10n, respectively, due to the length of their single words, using a capital L to distinguish the i in i18n and to make it easy to distinguish the lowercase l from the 1.(Wikipedia)

internationalization(i18N)

Problems to be solved for internationalization

The ability to display text in the user's native language.
Ability to enter text in the user's native language.
Ability to process text in the user's native language in a specific encoding.

Text Encoding

The Unicode character set can display almost every character known to man in code points ranging from 0 to 10FFFF (hexadecimal). It requires at least 21 bits for storage. The text encoding system UTF-8 adapts Unicode code points to a reasonable 8-bit data stream and is compatible with the ASCII data processing system.UTF stands for Unicode Transformation Format.

Since 2009, UTF-8 has been the dominant encoding form on the World Wide Web. As of November 2019, UTF-8 is used in 94.3% of all web pages (some of which are ASCII only, as it is a subset of UTF-8), and 96% of the top 1000 pages. Therefore, UTF-8 encoding is recommended for internationalization.

This article Internationalization of IT products is never enough to "support English" mentions that some GBK-encoded texts have many The text that "looks the same" is actually slightly different. However, in order to save space in Unicode, the same Code Point is assigned to them.

How can we distinguish these identical characters with the same code (displaying a character in a different glyph, i.e. the same character)? This requires the help of locale.

When calculating the number of Chinese characters, it is usually done by character form, i.e., simplified, traditional, variant, new, old, etc., of a character representing the same phonetic meaning. This way of counting is in fact counting variants. Therefore, the number of glyphs included in large dictionaries has long been wrongly regarded as the size of the Chinese character system.(Wikipedia)

locale

A locale is the language environment of the software at runtime, which includes Language, Territory and Codeset. A locale is written in the following format: Language[_Territory[. UTF8. In Linux, a locale consists of the following parts.

LC_COLLATE: Controls character sorting.
LC_CTYPE: controls the character handling function in handling upper and lower case or determining if it is a character.
LC_MESSAGES: format of prompt messages.
LC_MONETARY: format of currency.
LC_NUMERIC: the format of the number.
LC_TIME: the format of time.

If your locale is en_US.UTF8, you must change it to zh_CN.UTF8 to display Chinese correctly. All supported locales are stored in the /usr/share/locale directory of the macOS operating system.

Language and Country Codes

The same language may have some subtle differences in different countries and regions, for example, there are some differences between American English and British English. The same country may also have multiple languages, for example, China has simplified and traditional languages. In the introduction to locale above we saw the use of language_region to express the exact language of a country.

For countries and languages ISO has developed corresponding standard codes ISO 3166-1 and ISO 639-1.

The browser uses the language code to send the name of the language accepted by the browser in the Accept-Language HTTP header. For example: it, de-at, es, pt-br.

gettext

GNU gettext is the GNU Internationalization and Localization (i18n) library, which is often used to write multilingualization (M17N) programs. Many programming languages such as C, C++, Python, PHP, Rust, Elixir, etc. support the use of gettext from within the language.

The following is the flow of how Java calls gettext to complete internationalization.

xgettext scans the source code to extract the input strings for the i18n functions tr(), trc(), and trn() and creates a pot file containing all the source language strings. The object that the translator needs to work with is the .po file, which is generated by the msginit program from the .pot template file.
msgmerge merges the strings into a po file containing the translations for a single locale.
msgfmt is used to generate Java class files that inherit from the Java ResourceBundle class.

The following diagram shows the flow of internationalization in PHP using gettext.

Elixir implements i18n's directory structure using gettext.

priv/gettext
└─ en_US
|  └─ LC_MESSAGES
|     ├─ default.po
|     └─ errors.po
└─ it
   └─ LC_MESSAGES
      ├─ default.po
      └─ errors.po

Internationalization Process

The process of using gettext is a typical process of making an application support i18n internationalization.

Configure the i18n framework. i18n framework automatically obtains the relevant language files by the language identifier of the system or browser (in the case of web applications). For example, gettext uses a file with .mo suffix, while Javascript is usually a .json file and Java is a .properties file.
extract the hard-coded source language text. Call the i18n function at the hard-coded place. For this part can be extracted manually, or automatically through a program or plugin (e.g. i18next internationalization framework for Javascript has i18next-scanner).
finally implement localization. Translate (either manually or by machine translation, there are also relevant translation platforms that can be integrated) these extracted files in the language of the country to be supported.

Localization(L10N)

Localization Process

A typical localization flow chart is shown in the figure above. Among the parties involved are.

Dev Team: Developers make the system internationalized and deploy the machine-translated version of the multilingual version to the integration environment for testers to test, and can build an automated translation integration pipeline.
Market Team: Confirm the market and supported languages of the product, organize the glossary of terms involved in the product, and purchase professional translation services to determine the final multilingual translated version. In the case of large companies, there may be a dedicated globalization team to do this work.
Translation management platform (TMS): completes the management of translation languages, generally with specific API interfaces or SDK development kits that can be integrated into CI/CD environments and can automate the upload and download of source and translation language files. It also has a management interface for translators to modify and determine the final version of the translation. Can provide multiple machine translation services, as well as provide human translation for purchase or complete human translation in an open source collaborative manner.

Develop a localization strategy

Region and Language

This piece begins with these basic antecedent considerations.

Business implications of region
Default region of the user
Default language of the locale
Whether different regions use the same system
Whether to support users to switch locales
Whether the user can belong to multiple locales
Whether there is a one-to-one relationship between locale and country
Mapping between locale and language
Is there a linkage between locale and language (can the user see all languages supported by all locales)
Whether language switching needs to be saved to the user's personal information
Whether the user's default language needs to be set by the user's environment language identifier (OS or browser)
Whether the service is deployed in multiple geographies and whether the data is isolated in multiple geographies

Add new region/language/service

Whether the system can support the process of adding new regions and new regions
Whether the system can support the process of adding new languages and the process of adding new languages
The process of localization when adding subservices to the system

Incremental Localization

Localization process when new pages or components appear when localization is implemented

Management of translation

Whether you need a translation management platform (TMS)
Selection of Translation Management Platform
Integration of translation management platform
Whether to subscribe to professional translation services
Development of collaborative processes with translation teams

Localized implementation

Whether each service of the system is localized by its own development team
Whether there is a dedicated localization team to do localization
The collaboration mode between localization team and each service development team
- Whether localization is done by code Open PR
- How each service development team does incremental localization
- Synchronization of knowledge on localization among teams
Development of technical standards for localization and promotion within the organization
- Use of industry standard libraries (e.g. Unicode Common Locale Data Repository CLDR) for language-specific formats for dates, times, time zones, numbers, and currencies
- The locale identifier is in language_region format, e.g. en_US for United States English language.

Localized multilingual implementation

Distinguish multilingualism by subdomains (gTLDS) or ccTLDs (e.g. ccTLDs). For example: https://en.wikipedia.org/
Distinguish multilingualism by URL paths. For example: https://localizejs.com/de/
Multilingual by URL query parameters (not SEO friendly). e.g. https://locize.com/?lng=de
Multi-language by user language setting. E.g. https://myaccount.google.com/language
Multilingual by browser local storage. E.g. https://www.instagram.com/

The Challenges of Localization

The challenges of localization are mainly issues arising from differences in language, culture, writing habits and laws in different geographic areas, in the following categories.

Text encoding: For text in most Western European languages, ASCII character encoding is sufficient. However, languages that use non-Latin alphabets (e.g. Russian, Chinese, Hindi and Korean) require a larger character encoding, such as Unicode.
Singular and plural: Different languages have different forms of singular and plural. The plural is used to represent a number that is "not one". The singular and plural forms vary from language to language, with the most common plural form being used to represent numbers of two or greater. In some languages, it is also used to represent fractions, zeroes, negative numbers, or twos.
Image translation: Images with text need to be translated.
Dynamic data (data from the API): The data passed from the back-end to the front-end that is displayed on the interface needs to be localized. But it is a challenge to distinguish the source of this data, for example, although the data is sourced from the back-end, it may come from a database, it may come from a file, it may come from other internal services, or it may come from a third-party dependent service.
Icons: Some icons that are highly recognizable in one region may look completely unrecognizable to users in other regions or be something else.
Name/address: The order of the last name and first name, and the order in which addresses are written. For example, in Chinese, the last name comes first, then the first name.
Gender: Some languages such as French place a lot of emphasis on gender.
Phone: The format of phone calls varies from country to country.
Voice: Inappropriate voices or cues may be offensive, and some countries are sensitive to the gender of the voice.
Color: Colors and shades are associated with geographic regions or markets, for example, red in the U.S. indicates a decline and in the A-share indicates a rise.
Units of Measure
- Currency: Currency formatting must take into account the currency symbol, the position of the currency symbol and the position of the minus sign. Most currencies use the same decimal separator and thousands separator as the numbers in the regional or area setting. However, in some places this is not the case, for example in Switzerland, the decimal separator for the Swiss franc is a period.
- Date and time: The internationalization of date/time involves not only the geographical location (e.g. localized representation of calendar such as day of the week, month, etc.), but also the time zone (TimeZone, for UTC/GMT offsets). Time zones are not only geographically defined, but also politically defined. For example, China geographically spans 5 time zones, but only uses one unified time zone. Many other countries have "daylight saving time" and the difference between Berlin time and Beijing time is subject to change. Sometimes it is 7 hours (winter time), sometimes it is 6 hours (daylight saving time).
- Numbers: There are also differences in the way numbers are represented in different countries and regions. Factors that affect the representation of numbers include the representation of numeric characters, the representation of numeric symbols, the type of numbers, etc.
- Weight/length/physical units: Because of the differences in units, multiple geographical versions of the same set of data need to be converted.
- Business-related units of measurement: For example, different countries have different billing rules for their products. This requires business staff support to find out the corresponding position and give conversion rules.
Sentence length: German is usually longer than English, and Arabic requires more vertical space.
Writing direction: In many languages it is left to right, but in Hebrew and Arabic it is right to left and in some Asian languages it is vertical.
Punctuation: e.g. quotation marks ("") in English, low quotation marks (,,") in German and quotation marks (<<>>) in French.
Line breaks/splits: The rules of Asian CJK (Chinese, Japanese, Korean) character set languages are completely different from those of Western languages. For example, unlike most Western written languages, Chinese, Japanese, Korean and Thai do not necessarily use spaces to separate one word from the next. Thai does not even use punctuation.
Case conversion: English has case conversion, while Chinese has no case distinction.
Legal related: e.g. GDPR using personal data of EU citizens.
Politics-related: For example, localization involves the display of flags and maps, which can easily cause major accidents if not handled properly.
Sorting methods: For example, English is sorted alphabetically, while Chinese can be sorted in pinyin.

Do you need to consider SEO

If you are localizing a website for toC, you need to consider some things related to search engine optimization (SEO), such as this How to approach an international strategy mentions some key points.

If you offer your site in multiple languages, use a single language for content and navigation on each page, and avoid side-by-side translations.
Keep the content in each language on a separate URL and mark the language in the URL. For example, the URL www.mysite.com/de/ would tell the user that the page is in German.
Display the language you want to locate to Google via the hreflang meta tag. For example, <link rel="alternate" href="http://example.com" hreflang="en-us" />.
Do not translate only the template text, but also the content within the template.
do not use automatic translation exclusively, which can affect the user experience
don't use cookies or scripting techniques to switch languages, Google crawlers can't index this content properly.

Localization of product design

Using a more localized design for the same content in different geographies can lead to better results, as mentioned in the article Internationalization and Localization of Product Design about the different presentation of Spotify's song covers in different countries.

Localization under Microservices

The localization process for a single application is relatively simple from an architectural point of view. However, many applications nowadays are microservice architectures with multiple teams collaborating on the development model. If individual teams are responsible for the localization of their respective services, there must be a unified localization committee to develop technical standards for localization.

The identification of language markers.
The design of language switching in front and back-end solutions.
The design of translation automation process, etc.

Or there is a dedicated localization team to implement localization, and this team will be responsible for solving the previous problems. The project I am involved in falls into the latter category. Our team completed the localization of nearly a dozen microservice subsystems for the entire large system, and these dozens of systems were handled by several large groups of multiple teams, so the collaborative process of such cross-functional requirements (CFR) across multiple teams is a complex task.

Localized technical or business standards development

Prior to the implementation of localization, it is important to identify the relevant technical or operational standards, some of which are.

Internationalization implementation standards for different technology stacks in front and back ends. Since there may be multiple technology stacks in microservices, each with its own internationalization implementation, the development of implementation standards for different technology stacks can help to use the same implementation across services.
The determination of locale markers.
- The possibility of storing language-related text in front-end or back-end static text extraction to files named by language identifiers, e.g. en.json for static text in English, and en_US.json for text related to US English (e.g. units of measure, dates, numbers, currency, etc.).
- The language_region format is uniformly used in remote service calls (front-end calls to back-end or back-end calls to other internal or external services), e.g. en_US stands for getting the localized version of the English language for the US region.
Language specific formats for dates, times, time zones, numbers and currencies use industry standard libraries. For example, using libraries that implement the CLDR standard.
Dynamic data type identification. Identifying, for example, which data comes from internal systems (databases or files); which comes from external systems; whether these dynamic data have internationalization capabilities; and how to localize them in stages.
Localization of documents. Localization of electronic documents (PDF) or emails generated by the back-office system. If these documents are sent to clients, also consider whether to generate documents in the client's language preference.
List of supported regions and languages. For example, whether the error page is entered when an unsupported country or language appears or whether the default regional or language localized version is displayed.
The default region and default language.
whether the region and language have a binding relationship.
Whether the language switch needs to be saved to the user's personal information.

Development Environment and Business Processes

In fact, the most time-consuming part of localization for our team was the start-up of the local environment. With so many services involved and slight differences in the way different services are launched, and even wrong guidance documents, we needed to keep stepping on the toes to finish setting up the environment. In the end, our way of dealing with this was to contact the development teams, and each time we did the pre-localization of a service, we would ask the development team to help us set up the local environment.

Another difficulty was our lack of understanding of the business. Since each service has a large number of components and pages, including dynamic data from different sources of back-end services, it was hard to figure out just by ourselves. In the end, when we did the pre-localization of this service, we would get business analysts from the development team to help us introduce the business processes involved in this service.

Static text processing

Sorting through static text on the front and back ends to identify which systems have internationalization capabilities (initial language versions have had locale files extracted and internationalization libraries set up).
Identifying where date, currency, and number formats occur and calling the industry standard libraries identified by the localization technical standards in those places.
Identification of incremental static text translation processes, which need to be used to process new texts when they are added after the system has already been localized, using localization processes.
Automated integration of translation platforms, where development teams use scripts or CI/CD streams to automatically upload and download files in the original and translated languages.

Whether to store language and region settings

Some internationalized sites have language or region switching designed as hyperlinks that allow users to access different language and region versions of the site, which do not require storing language or region configurations.

Sites with user profile configuration generally offer to set the preferred language and region in the profile settings, so that users can synchronize the last set language or region when switching devices.

If your site users switch devices infrequently, a simple process can store these configurations in the browser store. When the user switches devices, the default settings are automatically restored. The advantage of such a design is that it is simple, and it is easier to overtake to other solutions later. The specific design chosen needs to be combined with the specific business to choose.

Localization of back-end services

Localization of back-end services involves the following four components.

Static text. These can be read by walking through the code to find the relevant string.
Databases, caches or files. Initial data that does not meet localization needs can be found by walking through the database initialization script, but for dynamically stored data, tables need to be designed to meet multilingual storage as well. For some resource files where translation is necessary there is also a need to provide multilingual versions and to adapt the code that uses the files.
Remote calls to other internal services (RPC). The locale markers for internal service calls are part of the localization technology standards developed. For example, the locale = en_US HTTP header can be used to request pages in English for the United States.
Generate documents (PDF or Email). Generated documents include the final language version of the template static text and dynamic data rendering. Especially when these documents and emails need to be sent to users, they need to be generated in the language that matches the user's language.

If the technology stack of the back-end service is different, the localization team also needs to summarize the internationalization process for the different technology stacks of the back-end service and synchronize it with other development teams within the organization.

Localization of third-party services and resources

There are cases of calling external services in the backend service remote call. If you call external services, you need to confirm whether the external services support multi-language version first, and if they do, you can integrate them according to the docking documentation. If not, you need to contact the external service provider to determine the support plan.

Release Process

Since the implementation of localization involves the transformation of more than a dozen subservices, localization can be controlled by Feature Toggles to be turned on or off in different environments. The tests affected by localization (unit tests, integration tests and UI tests) also need to be controlled via Feature Toggles so that the test suite of the original service is minimally affected.

Once all services have been localized and implemented, the localized Feature Toggles for all services can be opened to bring the final version online.

There are two designs to choose from regarding localized Feature Toggles.

Centralized Feature Toggles. for example, a centralized feature configuration service can be built, and all localization-related services request this service to get the configuration switch status. The advantage is that localized features can be switched on and off in real time without going back online. The downside is that there is no flexibility to control the localized features of each service.
Independent Feature Toggles: In contrast to centralized, each service sets its own localized Feature Toggles for flexible decoupling, but the downside is that each switch requires a new release to bring a single service online.

Localization under micro front-end architecture

As the above figure shows a micro front-end architecture website, the whole website interface is composed of five service pages of A/B/C/D/E. The language switch button is on service A. When the user switches from English to Chinese, the other services B/C/D/E need to switch their respective interfaces to the Chinese language version.

One way is to have the internationalization (i18n) library instance initialized by Service A and mounted on the browser window object when the browser loads the page, and Services B/C/D/E use the internationalization library instance object initialized by Service A. When switching languages, the internationalization instance object of Service A switches the language of all services.

The locale files for each service can be loaded into the browser uniformly by Service A. The advantage of this approach is that we know when the last language file is loaded, which means that the localization of all services on the whole page is initialized and the user can switch languages normally.

Localized testing

Localization testing verifies that the application or website content meets the language, cultural and geographical requirements of a particular country or region.

For more details, see this article Localization testing: why and how to do it.

Localization Platform

A very important piece of localization is the selection of a suitable translation management platform (TMS), which generally has the following function points.

Glossary: A glossary of specialized brand terms or domain terms that can help translators more accurately translate specialized language related to a product or market.
Translation Memory (TM): TM is a database for storing strings of previously translated content. Translations are reused for the same or similar content. This ensures consistency of translations.
Context Editor (In-Context Editor): This editor crawls website pages and allows translators to understand the context of the entire page, helping to improve the quality of the translation.
Machine Translations (Machine Translations): most TMS platforms are docked to some machine translation platforms (such as Google Translate), which can automatically translate the target language and are suitable for developers.
Human Translations: Professional human translation services can be ordered from the TMS platform. However, there are also features such as Crowdin, which provides localized translation collaboration for open source projects, and anyone can participate in this project for free, and the translated text with high votes will be used first.

Major localization platforms.

This concludes some introductions to internationalization and the basic process of localization. Localization is a complex task, and the biggest difficulty is not knowing enough about the target language and culture. But after you've read this article, I hope it will give you more confidence to do localization-related work.

Adventures in Serverless Application Development

Dawei Ma — Sun, 14 Feb 2021 08:13:51 +0000

Background

This article describes the entire process of developing an application based on AWS Serverless technology. The infrastructure is built using Serverless Framework and Terraform. The core module of the system is a timed task that is executed once a day: the task obtains a set of ETF index fund price data via Tushare , and after processing, generates a text of the trading signals and stores it in an S3 bucket. After that, a message is sent to the AWS SNS Topic, and users subscribed to the topic will receive an email alert. There is a user email subscription portal on the system's web page, and you can also access the daily history of trading signals.

This article covers the following:

Build a Docker image to execute Core Service.
The following AWS infrastructure services are required to build Core Service with Terraform.
- Set up an ECR repository.
- Set up an ECS cluster using Fargate.
- Set up Fargate tasks.
- Set up CloudWatch timed tasks.
- Set up IAM permission roles.
- Setting up SNS Topics.
- Setting up a VPC network.
The following AWS infrastructure services required to build API Services and the Web using the Serverless Framework.
- Set up Lambda functions.
- Set up the API Gateway.
- Setting up Route53.
- Setting up CloudFront.
- Setting up TLS certificates.
- Setting up S3 buckets.
- Setting up CloudFormation.

The final result is available at: Online version.

The source code is available at: Code Repository.

The technology stack is as follows:

Prerequisites

You need to register for the following account first:

Background Knowledge

Application Architecture Evolution History¹

Monolithic application (Monolithic): small applications suitable for startups; good performance.
Service-oriented (SOA): large applications suitable for complex enterprise business.
Microservices (Microservices): complex elastic scalable applications for experienced teams.
Serverless (Serverless): low cost, suitable for background tasks; also suitable for applications with a large number of customers and applications that grow rapidly and need to scale infinitely.

Serverless

Build and run applications without thinking about servers²

Serverless computing, or serverless for short, is an execution model in which a cloud provider (AWS, Azure, or Google Cloud) is responsible for executing a piece of code by dynamically allocating resources and charging only for the resources used to run the code. The code typically runs in a stateless container and can be triggered by a variety of events including HTTP requests, database events, queue services, monitoring alerts, file uploads, scheduling events (cron tasks), and more. The code that is sent to the cloud provider for execution is typically in the form of a function, and thus serverless computing is sometimes referred to as "function as a service" or FAAS.³

Advantages

No server management costs
Flexible scaling
Pay for service uptime
Self-contained high availability and fault tolerance

Disadvantages

Poor cold start performance
Complex monitoring and debugging
Dependence on cloud vendors

DevOps

This is a panorama of DevOps tools, and I have used only a small part of them in this application development process, even without testing process. Of course, the application of these tools also need to consider the actual situation of the project, flexible application.

Application Architecture

The application is divided into three main modules:

Core Service：Background timing tasks, providing the function of getting fund price data, analyzing and generating trading signals and sending emails.
API Service：Provides subscription theme API.
Web：Provides subscription theme entrance page and view historical trading signal records.

Coding Implementation

Since you'll be using a variety of AWS cloud services next, to learn about them see this article AWS Services Explained .

Code Structure

.
├── api # api service
│   ├── serverless.yml
│   └── sns.js # lambda function
├── core # core service
│   ├── Dockerfile
│   ├── Infrastructure
│   │   └── tf-fargate
│   │       ├── cloudwatch.tf
│   │       ├── ecr.tf
│   │       ├── ecs.tf
│   │       ├── iam.tf
│   │       ├── main.tf
│   │       ├── output.tf
│   │       ├── sns.tf
│   │       ├── tasks
│   │       │   └── task_definition.json
│   │       ├── variables.tf
│   │       └── vpc.tf
│   ├── Makefile # CLI entry
│   ├── requirements.txt
│   └── src
│       ├── fund.txt
│       └── main.py # fargate task
└── web # web service
    ├── binaryMimeTypes.js
    ├── client
    │   ├── assets
    │   │   └── styles
    │   │       └── global.less
    │   ├── components
    │   │   └── navbar.vue
    │   ├── layouts
    │   │   └── default.vue
    │   ├── pages
    │   │   └── index.vue
    │   └── plugins
    │       └── iview.js
    ├── index.js
    ├── nuxt.config.js
    ├── nuxt.js # lambda function
    ├── package-lock.json
    ├── package.json
    ├── secrets_example.json
    ├── serverless.yml
    └── yarn.lock

Core Service

Core Service runs through AWS Fargate, which is better suited than Lambda for running long background tasks. core Service is developed in Python, and to make it run in an AWS ECS environment, a Docker image is first built and then pushed to an AWS ECR repository.

Docker Image

FROM python:3.8-slim-buster

USER root
WORKDIR /tmp

# for source
RUN rm /bin/sh && ln -s /bin/bash /bin/sh

# for compile
RUN  apt-get update \
  && apt-get install -y wget \
  && apt-get install -y build-essential \
  && rm -rf /var/lib/apt/lists/*

# for TA-Lib
RUN pip install numpy && \
  wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz && \
  tar -xvzf ta-lib-0.4.0-src.tar.gz && \
  cd ta-lib/ && \
  ./configure --prefix=/usr && \
  make && \
  make install
RUN rm -R ta-lib ta-lib-0.4.0-src.tar.gz

# set the working directory in the container
WORKDIR /code
# copy the dependencies file to the working directory
COPY requirements.txt .
# install dependencies
RUN pip3 install -r requirements.txt
# copy the content of the local src directory to the working directory
COPY src/ .
# command to run on container start
CMD [ "python", "./main.py" ]

One thing to keep in mind here is the choice of base image, usually we use the alpine version when choosing a Python image. However, the alpine version requires a lot of compilation when installing some local binary packages, and this compilation will encounter various errors, so I finally chose the buster version, which is based on ubuntu. If you want to know more about it, see this article《Using Alpine can make Python Docker builds 50× slower》。

After obtaining the API Token for Tushare, you need to run Docker locally.

docker build -t invest-alchemy/core . # Build
docker run -t -i -e TUSHARE_API_TOKEN=xxxx invest-alchemy/core # Local Run

The ECR repository is then created in AWS, and the locally built images can then be pushed to the ECR for use by ECS tasks.

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin replace_with_your_ecr_addr.dkr.ecr.us-east-1.amazonaws.com # Login ECR

docker build -t invest-alchemy/core . # Local Build

docker tag invest-alchemy/core:latest replace_with_your_ecr_addr.dkr.ecr.us-east-1.amazonaws.com/invest-alchemy/core:latest # Add Tag

docker push replace_with_your_ecr_addr.dkr.ecr.us-east-1.amazonaws.com/invest-alchemy/core:latest # Push to Remote Repo

Infrastructure as Code

The next step is to build the required infrastructure (ECS/IAM/SNS/VPC/CloudWatch) via Terraform. The main reference for this piece is this《Serverless job scheduling using AWS Fargate》。

ECR/ECS/Task

See the source code for details, note that set capacity_provider to FARGATE_SPOT can significant cost reduction.

CloudWatch

See the source code for details, note that ecs_target/network_configuration can use the default VPC network, be sure to set the assign_public_ip to true, otherwise the container task will not be able to access the external network.

VPC

Use the AWS default VPC network. AWS Fargate can run in multiple network modes; the simplest, public subnet mode, is chosen here.

For other scenes see 《CloudFormation Templates for AWS Fargate deployments》.

For further study you can read this article 《Fargate networking 101》.

IAM

######################### Role used by the container regulates what AWS services the task has access to, e.g. your application is using a DynamoDB, then the task role must give the task access to Dynamo.
resource "aws_iam_role" "ecs_service_role" {
  name               = "${var.project}_ecs_service_role_${var.env}"
  assume_role_policy = "${data.aws_iam_policy_document.ecs_service_assume_role_policy.json}"
}

resource "aws_iam_role_policy" "ecs_service_policy" {
  name   = "${var.project}_ecs_service_role_policy_${var.env}"
  policy = "${data.aws_iam_policy_document.ecs_service_policy.json}"
  role   = "${aws_iam_role.ecs_service_role.id}"
}

data "aws_iam_policy_document" "ecs_service_policy" {
  statement {
    effect = "Allow"
    resources = ["*"]
    actions = [
        "iam:ListPolicies",
        "iam:GetPolicyVersion"
    ]
  }
}

data "aws_iam_policy_document" "ecs_service_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role_policy_attachment" "ecs_service_role_policy_attachment" {
  role       = aws_iam_role.ecs_service_role.name
  policy_arn = "arn:aws:iam::aws:policy/AWSLambdaFullAccess" # https://gist.github.com/gene1wood/55b358748be3c314f956
}

######################### Role used by the container enables the service to e.g. pull the image from ECR, spin up or deregister tasks etc

resource "aws_iam_role" "ecs_task_execution_role" {
  name = "${var.project}_ecs_task_execution_role_${var.env}"

  assume_role_policy = <<EOF
{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Action": "sts:AssumeRole",
     "Principal": {
       "Service": "ecs-tasks.amazonaws.com"
     },
     "Effect": "Allow",
     "Sid": ""
   }
 ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "ecs_task_execution_role_policy_attachment" {
  role       = aws_iam_role.ecs_task_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy" # https://gist.github.com/gene1wood/55b358748be3c314f956
}

######################### Role used for ECS Events

resource "aws_iam_role" "ecs_events_role" {
  name               = "${var.project}_ecs_events_role_${var.env}"
  assume_role_policy = "${data.aws_iam_policy_document.ecs_events_assume_role_policy.json}"
}

resource "aws_iam_role_policy_attachment" "ecs_events_role_policy" {
  policy_arn = "${data.aws_iam_policy.ecs_events_policy.arn}"
  role       = "${aws_iam_role.ecs_events_role.id}"
}

data "aws_iam_policy" "ecs_events_policy" {
  arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceEventsRole" # https://gist.github.com/gene1wood/55b358748be3c314f956
}

data "aws_iam_policy_document" "ecs_events_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type = "Service"
      identifiers = ["events.amazonaws.com"]
    }
  }
}

Three roles are defined here.

ecs_service_role: container application privileges, such as Core Service needs to push data to S3 and SNS, you need to have the relevant privileges for this role.
ecs_task_execution_role: ECS task execution permission, for example, if ECS needs to pull images from ECR, it needs to have the permission to access ECR.
ecs_events_role: CloudWatch timing task permission, such as timing tasks need to execute ECS tasks, you need to have the permission of AmazonEC2ContainerServiceEventsRole role.

Application Secrets

Applications always need to rely on some sensitive information, such as various api tokens. Core Service relies on Tushare API token, so it needs to be injected through Terraform. Here i refer to this 《A comprehensive guide to managing secrets in your Terraform code》.

The article provides these ways of managing sensitive information.

Environment variables
Encrypted files (AWS KMS)
Key repository (AWS Secrets manager)

The first way of environment variables is the simplest, and the next two have a certain cost of use, because the key security requirements are not high, the first simple way is used here.

First define the variables.

variable "TUSHARE_API_TOKEN" {
  description = "Tushare API Token from .env"
  type        = string
}

Then inject this environment variable at the ECS definition.

data "template_file" "task" {
  template = "${file("./Infrastructure/tf-fargate/tasks/task_definition.json")}"

  vars = {
    project             = "${var.project}"
    aws_region          = "${var.aws_region}"
    ecr_image_uri       = "${var.ecr_image_uri}"
    TUSHARE_API_TOKEN = "${var.TUSHARE_API_TOKEN}" # 注入变量
  }
}

Finally, inject this variable into the container at task_definition.json.

"environment": [{"name": "TUSHARE_API_TOKEN", "value": "${TUSHARE_API_TOKEN}"}]

This key needs to be entered each time a change is made, so that the key is not leaked to the code repository.

Make Build Script

################ Config ########################
S3_BUCKET ?= invest-alchemy
AWS_REGION ?= us-east-1
ENV ?= dev
ECR := 745121664662.dkr.ecr.us-east-1.amazonaws.com/invest-alchemy-core-ecr-dev # ECR Repository Example: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/{project_name}-ecr-{env}
################################################

################ Artifacts Bucket ##############
artifacts:
    @echo "Creation of artifacts bucket"
    @aws s3 mb s3://$(S3_BUCKET)
    @aws s3api put-bucket-encryption --bucket $(S3_BUCKET) \
        --server-side-encryption-configuration \
        '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'
    @aws s3api put-bucket-versioning --bucket $(S3_BUCKET) --versioning-configuration Status=Enabled
################################################


build-docker:
    @echo "run aws ecr get-login --region $(AWS_REGION) first"
    @docker build -t $(PROJECT) .
    @docker tag $(PROJECT) $(ECR)
    @docker push $(ECR)

################ Terraform #####################

init:
    @. .env
    @terraform init \
        -backend-config="bucket=$(S3_BUCKET)" \
        -backend-config="key=$(PROJECT)/terraform.tfstate" \
        ./Infrastructure/tf-fargate/

validate:
    @terraform validate ./Infrastructure/tf-fargate/

plan:
    @terraform plan \
        -var="env=$(ENV)" \
        -var="project=$(PROJECT)" \
        -var="description=$(DESCRIPTION)" \
        -var="aws_region=$(AWS_REGION)" \
        -var="artifacts_bucket=$(S3_BUCKET)" \
        ./Infrastructure/tf-fargate/

apply:
    @terraform apply \
        -var="env=$(ENV)" \
        -var="project=$(PROJECT)" \
        -var="description=$(DESCRIPTION)" \
        -compact-warnings ./Infrastructure/tf-fargate/

destroy:
    @read -p "Are you sure that you want to destroy: '$(PROJECT)-$(ENV)-$(AWS_REGION)'? [yes/N]: " sure && [ $${sure:-N} = 'yes' ]
    @terraform destroy ./Infrastructure/tf-fargate/

First execute make build-docker to build the image and upload it to ECR, then execute make init to initialize Terraform, then execute make validate && make plan to verify that there are no problems with the infrastructure configuration. If there are no problems, run make apply to build the real infrastructure.

Workflow

If there are changes to the system code, the following process can be repeated.

Modify the code
make build-docker
make apply

API Service

The API Service has only one API for subscribing to SNS topics, which is used to help users subscribe to the topics provided by Core Service.

Serverless Framework

service: api
app: invest-alchemy
org: madawei2699

provider:
  name: aws
  role: arn:aws:iam::745121664662:role/invest-alchemy-lambda
  runtime: nodejs12.x
  stage: dev
  region: us-east-1

functions:
  subscribe_sns:
    handler: sns.subscribe_sns
    memorySize: 128
    description: Subscribe sns topic.
    environment:
      snsTopicArn: arn:aws:sns:us-east-1:745121664662:trade-signal-topic
    events:
      - http: 
          path: subscribe
          method: post
          cors: true
          integration: LAMBDA

The function subscribe_sns is defined here, and we implement it with Javascript, the source code is available at sns.js, so we won't go over it here.

Note that the code needs to define the SNS topic, so it needs to have the role permission to subscribe to SNS, which we specify in the Serverless configuration as arn:aws:iam::745121664662:role/invest-alchemy-lambda, which has permission to subscribe to the SNS, with the additional policies AWSLambdaBasicExecutionRole and AmazonSNSFullAccess.

Code executed in Lambda can be imported directly into the AWS SDK without having to install it, and there is no need to set AWS Credentials, as the function executes with the permissions attached to the role directly.

Web

The web UI provides user-accessible pages for users to sign up for email subscriptions. Vue.js and Nuxt.js are used in this module to build SEO-friendly Server-Side Rendered web pages. I mainly refer to this article AWS | Vue Nuxt Ssr .

service: web
app: invest-alchemy
org: madawei2699

provider:
  name: aws
  runtime: nodejs12.x
  stage: ${self:custom.secrets.NODE_ENV}
  region: us-east-1
  environment: 
    NODE_ENV: ${self:custom.secrets.NODE_ENV}

functions:
  nuxt:
    handler: index.nuxt
    memorySize: 256
    events:
      - http:
          path: /
          method: any
          cors: true
      - http:
          path: /{proxy+} 
          method: any
          cors: true

plugins:
  - serverless-apigw-binary
  - serverless-domain-manager
  - serverless-offline

custom:
  secrets: ${file(secrets.json)}
  apigwBinary:
    types:
      - '*/*'
  customDomain:
    domainName: ${self:custom.secrets.DOMAIN}
    basePath: ''
    stage: ${self:custom.secrets.NODE_ENV}
    createRoute53Record: true

Note here that the gateway integrates custom domain names and TLS certificates: you need to first configure the domain name in AWS Route53 and then apply for a TLS certificate in AWS Certificate Manager, please refer to the article mentioned above for the detailed process.

Once the certificate application is successful, you can execute sls create_domain to create the DNS information for the domain name.

Finally, run npm run deploy to deploy to AWS. If you want to debug locally, you can run npm run start-server.

Debugging / Logging

Select the CloudWatch service in the AWS console to view the log group and locate the issue by analyzing the associated logs. If no logs are generated, you can also see if timed task events are generated by viewing the logs in CloudTrail.

Issues That Need Attention

Serverless Costing

One of the major advantages of Serverless is that it is pay-as-you-go, which is cheaper than buying a separate VPS for timed tasks or low traffic sites. It also has extremely high availability and elastic scalability, which is not possible with a single VPS.

To analyze the cost, you can use the AWS Billing service. The billing services we use (excluding some services that have negligible billing on the system such as S3/VPC/CloudFront, etc.) include

API Gateway: $1 per 1 million requests.
ECS Spot Fargate: $0.01289974 per vCPU/hour and $0.00141649 per GB of memory/hour.
Lambda: $0.0000002083 per 128M RAM/100 ms.
SNS: $2 per 100,000 email pushes.

All of the above are based on the US East region.

We know from the CloudWatch log set that Lambda execution time per request is 1200 ms per run of the API and Web service; Core service runs 3 minutes per day. The cost of Lambda is: 200000 * 1200 / 100 * 0.0000002083 = $0.5; ECS cost is 3 * 30 / 60 * (0.01289974 + 0.00141649) + 0.00141649) = $0.02. API Gateway costs 200000 / 1000000 = $0.2; SNS assuming 1,000 people subscribe to the email is 1000 * 30 / 100000 = $0.3.

Then the monthly cost is: 0.5 + 0.02 + 0.2 + 0.3 = $1.02. Take vultr VPS for example, the cheapest configuration with 1 vCPU + 512 MB configuration is $2.50 a month, and that doesn't include the cost of sending emails.

Cold Starts

The background timing tasks are not sensitive to the start time, if you are very concerned about the start time can be constantly warmed up by some timing tasks, but this will also lead to higher costs. For further information, please refer to this article 《Solving Serverless Cold Starts with Advanced Tooling》。

VPC Price

Most of the VPC services are not charged, except VPN/NAT Gateway/Endpoints (Gateway is free/Interface is charged), while the Endpoints of Interface are charged very expensive and should be stopped as much as possible when they are not needed, and you have to remove the related AWS services to stop charging.

Note that Endpoints are charged based on available zones, so if an Endpoint is created with 6 available zones, it will be charged 6 times as much.