DEV Community: David Hwang

6/12 TIL: AWS solutions architect cert S3 part 2

David Hwang — Sun, 13 Jun 2021 12:00:30 +0000

Encryption
- Encryption in Transit
  - SSL/TLS (https)
- Encryption at Rest (data being stored)
  - Server side: Amazon helps you encrypt the object
    - S3 Managed Keys - SSE-S3
    - AWS Key Management Serviced, Managed Keys - SSE-KMS
    - Server Side Encryption With Customer Provided Keys - SSE-C
  - Client side: you encrypt the object and upload it to S3
Versioning
- Stores all versions of an object (including all writes and even if you delete an object)
- Great backup tool
- Once enabled, Versioning cannot be disabled, only suspended
- Integrates with Lifecycle rules
- Versioning's MFA Delete capability, which uses multi-factor authentication, can be used to provide an additional layer of security
- Making one version of object public **doesn't* make other (previous or new) objects public; you have to configure them individually
- *Deleting an object will create Delete Marker over it and hide all the previous versions; deleting the Delete Marker will restore the object
Lifecycle Management with S3
- Go into the Bucket > Management tab > Create Lifecycle rule
- Automates moving your objects between different storage tiers or permanently delete previous versions (configure the timeline)
- Can be applied to current and previous versions
S3 Object Lock & Glacier Vault Lock
- S3 Object Lock
  - Stores objects using a write once, read many (WORM) model. It can help you prevent objects from being deleted or modified for a fixed amount of time or indefinitely
  - Used to meet regulatory requirements that require WORM storage, or add an extra layer of protection against object changes and deletion
  - Can be on individual objects or applied across the bucket as a whole
  - Modes
    - Governance mode: users can't overwrite or delete an object version or alter its lock settings; you can still grant some users permission to alter the retention settings or delete an object if necessary
    - Compliance mode: a protected object version can't be overwritten or deleted by any user, including the root user; for the duration of the retention period
      - Retention Periods: protects an object version for a fixed amount of time
      - Legal Holds: works like Retention Periods, only that it remains in effect until the s3:PutObjectLegalHold permission is removed
- S3 Glacier Vault Lock
  - Allows you to easily deploy and enforce compliance controls for individual S3 Glacier vaults with a Vault Lock policy. You can specify controls such as WORM in a Vault Lock policy and lock the policy from future edits.
S3 Performance
- S3 Prefix
  - mybucketname/folder1/subfolder1/myfile.jpg
  - prefix is the middle bit between the bucket name and the object
  - S3 has extremely low latency. You can get the first byte out of S3 within 100-200ms
  - You can achieve 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEADER requests per second per prefix
  - You can get better performance by spreading your reads across different prefixes. For example, if you are using two prefixes, you can achieve 11,000 requests per second.
  - S3 limitations when using SSE-KMS as your encryption
  - When you upload a file, you will call GenerateDataKey in the KMS API
  - When you download a file, you will call Decrypt in the KMS API
  - Uploading/downloading will count toward the KMS quota (region-specific, it's either 5,500, 10,000, or 30,000 requests per second)
- Multipart uploads
  - Recommended for uploading files over 100MB, required for files over 5GB
  - Parallelize uploads (increase efficiency)
- S3 Byte-Range Fetches
  - Parallelize downloads by specifying byte ranges
  - Speed up downloads or download partial amounts of the file
S3 Select & Glacier Select
- S3 Select
  - Enables applications to retrieve only a subset of data from an object by using simple SQL expressions. You can achieve drastic performance increases—up to 400% improvement
  - Without S3 Select, you would need to download, decompress, and process the entire CSV to get the data you needed
- Glacier Select
  - Some companies in highly regulated industries—e.g. financial services, healthcare, and others—write data directly to Amazon Glacier to satisfy compliance needs like SEC Rule 17a-4 or HIPAA. Many S3 users have lifecycle policies designed to save on storage costs by moving their data into Glacier when they no longer need to access it on a regular basis

6/10 TIL: AWS solutions architect cert S3 Part 1

David Hwang — Fri, 11 Jun 2021 10:59:11 +0000

Basics of S3
- S3 is a universal namespace; bucket names must be unique globally
- When you upload a file to S3, you will receive a HTTP 200 code if the upload was successful
- S3 is object based—i.e. allows you to upload files
  - Consists of:
    - Key (name)
    - Value (data)
    - Version ID
    - Metadata
    - Subresources (Access Control Lists, Torrent)
- Files can be from 0 bytes to 5TB; there is unlimited storage
- Files are stored in Buckets (equivalent to folders)
Data consistency in S3
- Read after Write consistency for PUTS of new objects (you are able to read the file right after writing to it; http PUTS)
- Eventual Consistency for overwrite PUTS and DELETES (latest version will always be used)
S3 guarantees:
- 99.99% availability for the S3 platform
- Amazon Guarantee 99.9% availability
- 99.999999999% durability for S3 information (11 x 9s)
S3 features:
- Tiered Storage available
- Lifecycle Management
- Versioning
- Encryption
- MFA Delete
S3 Storage Classes (Tiers)
1. S3 Standard
  - 99.99% availability, 99.999999999% durability, stored redundantly across multiple devices in multiple facilities, designed to sustain the loss of 2 facilities concurrently
2. S3 - IA (Infrequently Accessed)
  - For data that is accessed infrequently, but requires rapid access when needed
3. S3 One Zone - IA
  - When you want a lower cost option for IA data, but not require the multiple AZ data resilience
4. S3 - Intelligent Tiering
  - Automatically moving data to the most cost-effective access tier using ML
5. S3 Glacier
  - Secure, durable, and low-cost storage for data archiving; retrieval times configurable from minutes to hours
6. S3 Glacier Deep Archive
  - Lowest-cost storage class where a retrieval time of 12 hours is acceptable
S3 charged based on:
- Storage
- Requests
- Storage Management Pricing (different tiers)
- Data Transfer Pricing
- Transfer Acceleration
  - Enables fast, easy, and secure transfer of files over long distances between your end users and an S3 bucket. It takes advantage of Amazon CloudFront's globally distributed edge locations. As the data arrives at an edge location, data is routed to Amazon S3 over an optimized network path
  - If users want to upload a large file to a bucket in London, they can upload it to an edge location and those edge locations will use amazon backbone networks to bring it into London
- Cross Region Replication
  - Automatically replicates objects in a bucket across different regions

6/1 TIL: Unicode utf8mb4, sql COLLATE

David Hwang — Wed, 02 Jun 2021 03:51:44 +0000

Unicode

A character set that defines different character encodings, like UTF-8, UTF-16, and UTF-32

UTF-8

encodes common ASCII characters using 8-bits by assigning every character a unique number called a code point
vs. utf8mb4: a preferred alternative since it stores a maximum of four bytes per code point instead of 3 (utf8 is an alias of utf8mb3)—meaning utf-8 might not support some characters from other languages and symbols

An example of a sql file with the utf8mb4 charset


CREATE TABLE IF NOT EXISTS dbName.tableName

(

`id` int NOT NULL AUTO_INCREMENT,

`email`          varchar(40) COLLATE utf8mb4_unicode_ci NOT NULL,

`username`       varchar(20) COLLATE utf8mb4_unicode_ci  NOT NULL,

`password`       varchar(200) COLLATE utf8mb4_unicode_ci DEFAULT NULL,

PRIMARY KEY (`id`),

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Collation provides the sorting rules, case, and accent sensitivity properties for the data
You specify the COLLATE where ci stands for 'case insensitive'

5/31 TIL: CGO & GOOS in Go, Github Issue Template, Mathematical Induction

David Hwang — Tue, 01 Jun 2021 03:50:55 +0000

CGO & GOOS in Go

Cgo enables the creation of Go packages that call C code
Set CGO_ENABLED=0 or 1 along with Go build to disable or enable the cgo tool
You configure the Go Operating System using GOOS
Example of a go build within a Dockerfile:


RUN CGO_ENABLED=0 GOOS=linux go build -o fileName .

Github Issue Template

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository

Go Vendor

https://blog.gopheracademy.com/advent-2015/vendor-folder/

—Discrete Math—

Quotient-Remainder Theorem:

For all n ∈ ℤ, d ∈ ℤ+, there exists q,r ∈ ℤ so that n = dq + r with 0 ≤ r < d

Modular Arithmetic

n mod d; system of arithmetic where we are concerned with the remainder
Ex) Clock (n/12 = w + r), Day of the Week (n/7 = w + r)

Sequence

informal: an infinite ordered list of objects
formal: a function f: ℤ+ → C

Mathematical Induction

Prove: P(n), ∀n ≥ a
Step 1: Prove that P(a) is true—basis step
Step 2: Assume P(k) is true, prove P(k+1) is true (for k ≥ a )—induction step

5/30 TIL: mysql Docker image, Unix Domain Socket

David Hwang — Mon, 31 May 2021 03:24:21 +0000

Docker Compose

Do a 'docker-compose -f nameOfYourDockerComposeFile build' if you need your changes in Dockerfile to be reflected—because simply doing 'docker-compose up' doesn't rebuild an image

Docker mysql image

Configuration options can be passed as flags in the command line. Ex) --character-set-server=utf8mb4
When a mysql container is started for the first time, it will execute files with extensions .sh, .sql and .sql.gz that are found in /docker-entrypoint-initdb.d. So we can populate our mysql services by mounting a SQL dump into that directory. In our example below, we would store our .sql files in db/init and mount it to /docker-entrypoint-initdb.d under volumes:

services:
  db:
    container_name: mysql
    image: mysql
    restart: always
    environment:
      MYSQL_USER: user1
      MYSQL_PASSWORD: test
      MYSQL_ROOT_PASSWORD: test
      MYSQL_DATABASE: users
    ports:
      - '3306:3306'
    volumes:
      - mysql_db:/var/lib/mysql
      - ./db/init:/docker-entrypoint-initdb.d
    command:
      [
        'mysqld',
        '--character-set-server=utf8mb4',
        '--collation-server=utf8mb4_unicode_ci',
      ]
volumes:
mysql_db:

https://hub.docker.com/_/mysql

Unix Domain Socket

an inter-process communication mechanism that allows bidirectional data exchange between processes on the "same" machine—faster and lighter than IP sockets because it skips checks and operations like routing
vs. IP sockets: a mechanism allowing communication between processes over the network
https://serverfault.com/questions/124517/what-is-the-difference-between-unix-sockets-and-tcp-ip-sockets
blog post: https://medium.com/swlh/getting-started-with-unix-domain-sockets-4472c0db4eb1

5/28 TIL: Docker Compose, Volumes, cURL

David Hwang — Sat, 29 May 2021 02:43:01 +0000

Docker-compose


version: '3'
  services:
    server:
      build: .
      volumes:
        - .:/app
      ports:
        - 9090:9090
      command: ['wait-for-it.sh', 'db:3306', '--', 'air']
      depends_on:
        - db
    db:
      image: mysql
      restart: always
      environment:
        MYSQL_USER: admin
        MYSQL_PASSWORD: password
        MYSQL_ROOT_PASSWORD: password
        MYSQL_DATABASE: sampleDb
      ports:
        - '3306:3306'
      volumes:
        - mysql_db:/var/lib/mysql
      command:
        [
          'mysqld',
          '--character-set-server=utf8mb4',
          '--collation-server=utf8mb4_unicode_ci',
        ]
  volumes:
    mysql_db:

Two containers: server & db
server
- builds the Dockerfile in the current path
- runs on port 9090 on both the host and within the container
- executes 'wait-for-it.sh' to wait on the availability of port 3306 before running 'air'--using wait-for-it.sh is just an example and this shell script has to be installed as part of the Dockerfilehttps://github.com/vishnubob/wait-for-it
- depends on db container
db
- runs on mysql image
- we have a 'named volume' called mysql_db at the last line of this docker-compose.yml file and we have that mounted to 'var/lib/mysql' directory in the db container's virtual file systems (watch 3:40 here for details https://www.youtube.com/watch?v=p2PH_YPCsis&ab_channel=TechWorldwithNanaTechWorldwithNana)

cURL & its flags

curl is a tool to transfer data from or to a server, using one of the supported protocols (FTP, HTTP, SFTP, and many more)
Example:


RUN curl -fLo [install.sh](http://install.sh/) [https://raw.githubusercontent.com/cosmtrek/air/master/install.sh](https://raw.githubusercontent.com/cosmtrek/air/master/install.sh)

-f allows the HTTP to fail silently on server errors
-L will make curl redo the request on the new place if the server reports that the requested page has moved to a different location
-o writes the output to the specified file
-s is silent mode
-S is used with -s and it makes curl show an error message if it fails. ex) -sSfL

5/26 TIL: Dockerfile, apt, C++ Generics, ADT in C

David Hwang — Thu, 27 May 2021 01:59:28 +0000

--Docker--
Dockerfile for a Go project


FROM golang:1.15 AS builder

RUN apt update && apt upgrade -y && \

apt install -y git \

WORKDIR /app

# Get install script, run, and make binary

RUN curl -fLo install.sh https://raw.githubusercontent.com/cosmtrek/air/master/install.sh \

&& chmod +x install.sh && sh install.sh && cp ./bin/air /bin/air

Always combine RUN apt-get update with apt-get install in the same RUN statement as suggested in the Dockerfile best practices under the 'apt-get' section here: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
The WORKDIR command is used to define the working directory of a Docker container. Any RUN, CMD, ADD, COPY, or ENTRYPOINT command will be executed in the specified working directory.

apt vs. apt-get

apt is a subset of apt-get and apt is usually preferred
apt provides all the necessary commands for package management so just go with that

apt update vs. apt upgrade

apt update updates the list of available packages and their versions, but it does not install or upgrade any packages.
apt upgrade actually installs newer versions of the packages you have. After updating the lists
-y flag means "say yes to all"
https://askubuntu.com/questions/94102/what-is-the-difference-between-apt-get-update-and-upgrade

--C++ basics--

Generics in C++


#include <iostream>

using namespace std;

template <class T>

T GetMax (T a, T b) {

T result;

result = (a>b)? a : b;

return (result);

}

int main () {

int i=5, j=6, k;

long l=10, m=5, n;

k=GetMax<int>(i,j);

n=GetMax<long>(l,m);

cout << k << endl;

cout << n << endl;

return 0;

}

Templates in C++: https://www.cplusplus.com/doc/oldtutorial/templates/

Logical data structures—implemented using physical data structures: arrays or linked lists

Stack (linear, LIFO)
Queues (linear, FIFO)
Trees (non-linear)
Graph (non-linear)
Hash Table (tabular)

Abstract Data Type (ADT)

ADT is a type (or class) for objects whose behavior is defined by a set of value and a set of operations
The definition of ADT only describes what operations are to be performed but not how these operations will be implemented, making it abstract
List, Stack, & Queue ADTs: https://www.geeksforgeeks.org/abstract-data-types/

--Discrete Math--

Math Theorem Example: an even integer plus an odd integer is another odd integer

Proof:

Suppose m is even and n is odd (1. State the assumptions)
∃k1 ∈ Z and ∃k2 ∈ Z so that m = 2k1 and n=2k2 + 1 (2. Formally define the assumptions)
Then, m + n = (2k1) + (2k2+1) = 2(k1+k2) + 1. Let k3 = k1 + k2, and note it is an integer. (3. Manipulation)
Hence ∃k3 ∈ Z so that m + n = 2k3 + 1 (4. Arrive at definition of conclusion)
Thus m + n is odd (5. Conclusion)

Divisibility

For n and d integers, d ≠ 0, d | n ↔ if ∃k ∈ Z such that n = dk
When we say d | n, it means n is divisible by d

∀x ∈ D, P(x) → Q(x) where we quantify with specific x values, it uses a single arrow to show implication between two statements

P(x) ⇒ Q(x) with a double arrow is an implication between two predicates; it means the same thing as above

Other Proof Methods:

Disproving with Counterexamples
Proof by division into cases
Proof by contradiction
Proof by contrapositive

5/25 TIL: C++ basics (heap memory, referencing, pointer to struct)

David Hwang — Wed, 26 May 2021 03:01:13 +0000

Heap Memory in C

The heap is a large pool of memory that can be used dynamically - it is also known as the 'free store'. This is memory that is not automatically managed and you have to explicitly allocate & deallocate. It is where global variables are stored and you have whats called 'memory leak' if you fail to free/deallocate the memory.
To assign memory in heap, we use malloc()
You must delete (release/de-allocate) the memory in whenever you are dynamically allocating memory Ex) free(p) in C and delete [] p in C++ where p is an array


// allocates 20 bytes (5 * 4) of heap memory holding an int and assign that pointer to p

p = (int *)malloc(5 * sizeof(int));

free(p);

// C++

p = new int[5];

delete [] p;

Every pointer takes 8 bytes of memory regardless of its data type

Referencing in C++ (only C++)

Nickname/alias given to a variable


int a = 10;

int &r = a;

Reference has to be initialized (it cannot just be declared)
Now the variable a is also "called" r (this is not related to pointer)
Useful in parameter passing

Ex)


// instead of having to pass pointers 

void swap(int *x, int *y) { ... }

int main() {

int a = 10; b = 20;

swap(&a, &b);

}

// we can use referencing; now we don't have to pass the pointers

void swap(int &x, int &y) { ... }

int main() {

int a = 10; b = 20;

swap(a, b);

}

Pointer to Struct


struct userInfo u={'david', 25};

struct userInfo *p = &u;

// this won't work bc the dot takes precedence

*p.name = 'davidd';

// instead wrap p inside brackets

(*p).name = 'davidd';

// OR

p → name = 'davidd';

Create an object dynamically in heap using pointer


// malloc returns a void pointer; so we typecast it 

p = (struct userInfo*) malloc(sizeof(struct userInfo));

Array as Parameter

Arrays can only be passed by address; it cannot be passed by value at all (both C & C++)


// when we receive an array with square brackets, it can only be a pointer to the array 

void funcOne(int A[]) { ... }

// for this we have a pointer to an integer; it can point to any integer and even the array

void funcOne(int *A) { ... }

5/24 TIL: Predicates, Quantifiers

David Hwang — Tue, 25 May 2021 04:23:14 +0000

Predicates

A statement is either TRUE or false. But x > 5 is not a statement until we have the value determined for x.
A predicate is a sentence depending on variables which becomes a statement upon substituting values in the domain

Truth Set of a predicate P(x):

{ x ∈ D | P(x) }

shows which of the elements in the domain make the predicate true

Universal Quantifier ∀

"for all" (every)
main use: "quantifying" predicates
∀x ∈ D, P(x) — for all x in the domain, P(x) is true

Existential Quantifier ∃

"there exists" (some)
∃x ∈ D, P(x) — there exists x in the domain, such that P(x) is true

Universal/existential quantifiers turn predicates back into a statement

Negating a universal

~(∀x ∈ D, P(x)) ≡ ∃x ∈ D, ~P(x)
meaning: if it's not the case that for all values some properties are true, there is at least one value where the property is false

Negating logical statements with multiple quantifiers

∀x ∈ D, P(x)) ≡ ∃

Ex) "Every integer has a larger integer"
∀x ∈ Z, P(x) → ∀x ∈ Z, ∃y ∈ Z, y > x -negate→ ∃x ∈ Z, ∀y ∈ Z, y ≤ x

5/23 TIL: Go Signals, http, & Linux file systems

David Hwang — Mon, 24 May 2021 02:43:36 +0000

--Golang--
Go flag


addr = flag.String("addr", "0.0.0.0:9090", "Default HTTP address")

port 9090 will be the default value unless the user specifies the port on the command line by doing something like --addr 0.0.0.0:8080.
the first parameter is the name of the flag, the second parameter the default value, and the last parameter describes the default value

Package http


s := &http.Server{

Addr:           ":8080",

Handler:        myHandler,

ReadTimeout:    10 * time.Second,

WriteTimeout:   10 * time.Second,

MaxHeaderBytes: 1 << 20,

}

s.ListenAndServe()

https://golang.org/pkg/net/http/#Handler

Go Signals


sigChan := make(chan os.Signal)

// relays incoming signals to sigChan

signal.Notify(sigChan, os.Interrupt, os.Kill)

sig := ← sigChan

log.Println("Received terminate signal, gracefully shutting down", sig)

--Systems--
File Systems

Devices are represented by entries in the /dev directories
Each device has a corresponding device driver which implements a standard set of operations: open, read, write, close functions that were discussed in file IO topic
A device may have a corresponding hardware device driver that implements an API
A hard disk is divided into one or more partitions, each of which may contain a file system
A file system is an organized collection of regular files and directories
Linux implements file systems such as the ext2 file system
All the file systems in Linux are mounted under a single directory tree with a directory / at its root; a privileged process can mount and unmount a file system using mount() and unmount()
Information about the mounted file systems can be retrieved using statvfs()

--Discrete Math--
Modus Ponens

A valid argument is a list of premises from which the conclusion follows
Ex) if p, then q. p. Therefore, q.

Modus Tollens

If p, then q. ~q. Therefore, ~p.

Logical Arguments:

Generalization: p. Therefore, p V q. (always true)
Specialization: p ʌ q. Therefore, p.
Contradiction: ~p → c. Therefore, p.

5/22 TIL: Processes, Contrapositive, etc.

David Hwang — Sun, 23 May 2021 02:55:21 +0000

--Systems--

Processes

A process is an instance of an executing program; each process has a unique process id & its parent process id
Virtual memory of a process is logically divided into:
- Text initialized & uninitialized
- Data
- Stack - a series of frames with a new frame being added as a function is invoked & removed when the function returns; each frame contains local variables, function arguments, and called linkage information for a single function implication
- Heap
Command line arguments supplied when a program is invoked are made available via argc and argv arguments to the main function


#include<unistd.h>

int proc_id = getpid();

int par_proc_id = getppid();

Memory Allocation


malloc() // memory allocation

calloc() // contiguous allocation

free() // de-allocate

realloc() // re-allocate

// example

int* ptr

int num=8;

ptr = (int*)malloc(num*sizeof(int))

System Limits

SUSV3 (Single Unix Specification V3) adds a lot of new functions into C/C++ library
SUSV3 specifies limits that an implementations may enforce and system options that an implementation may support


#include <limits.h>

LONG_MIN, LONG_MAX, CHAR_MIN, CHAR_MAX // etc

--Discrete Math--

if p then q (p → q)

p → q ≡ ~p V q

~(p → q) ≡ ~(~p V q) ≡ (~~p ʌ ~q) ≡ p ʌ ~q

apply DeMorgan's Law
the only scenario where p → q is false is when p is T and q is F. Negating that p → q would be F. So this is equivalent to p ʌ ~q where p is T and q is F.

Contrapositive of a conditional

p → q ≡ ~q → ~p because

p → q ≡ ~p V q and
~q → ~p ≡ q V ~p
with ~p V q ≡ q V ~p being true

Converse: p → q is the statement q → p (not logically equivalent)

Inverse: p → q is the statement ~p → ~q (contrapositive of a converse)

conclusion: converse ≡ inverse since contrapositive statements are logically equivalent

Biconditional p ↔ q

a conjunctive statement where we have both p → q and q → p implications
"if and only if"

5/21 TIL: C/C++ setup on MacOS & C pointers

David Hwang — Sat, 22 May 2021 03:41:26 +0000

To run C programs, we need a compiler
- clang and gcc are usually used on MacOS—they are built-in
- we can check if these compilers exist in our system by doing this on terminal:
```
  gcc -v

  clang -v
```
- if they aren't found, we can install it by running: xcode-select --install
- https://www.youtube.com/watch?v=f0vVV4NPmjQ
To run a C program in VS code
- Install the 'C/C++' and 'code runner' extensions
- Write your C program and compile it by selecting Terminal > Run Build Task (choose gcc/clang build in the dropdown)

Pointers in C

https://www.cs.fsu.edu/~myers/c++/notes/pointers1.html
int *p, int* p are the same
asterisk (*) used in the declaration statement is used to define the pointer variable
asterisk (*) used elsewhere is used to dereference the pointer; for instance:


  int x = 10;

  // * used here in the declaration statement has nothing to do with dereferencing; we are simply declaring a variable named aptr of type 'pointer to int' (meaning it stores the address of an integer variable)

  int *aptr = &x; 

  printf("%x returns a memory address", aptr);

  // * here is now used for dereferencing

  printf("%d is the value dereferenced, *aptr);