DEV Community: Gavin Dang

AWS SQS

Gavin Dang — Thu, 05 Sep 2024 22:44:04 +0000

Amazon Simple Queue Service (SQS) is a distributed message queuing service introduced by Amazon Web Services (AWS) in 2004. It allows you to send and receive messages through an HTTP API over the internet. Designed with high scalability in mind, SQS facilitates seamless message exchange between different components of a system.

Key Features of SQS:

High Scalability: SQS scales automatically based on message read/write capacity, eliminating the need to manage queue scaling manually—AWS handles this for you.
Pay-as-You-Go Pricing: You only pay for the number of read/write operations you perform, with no recurring fees.
Easy Setup: There is no need to set up any infrastructure to use SQS. You simply call the API to read or write messages.
Standard and FIFO Queues: SQS offers two types of queues: Standard and FIFO. You can choose the type that best suits your needs.
Automatic Deduplication in FIFO Queues: SQS automatically handles duplicate messages in FIFO queues. This ensures that tasks are executed only once, making FIFO queues ideal for processes where each task should be executed exactly once.
Error Handling with Dead Letter Queues (DLQs): A critical feature for debugging, DLQs store messages that could not be processed successfully. You can inspect and handle failed messages separately from the main queue.

How Does SQS Work?

Behind the Scenes:

Queue Operation: Typically, a queue operates by having producers send messages to it. Consumers then retrieve and process these messages. The system tracks messages as pending or in-flight and delivers them to consumers. Background jobs at the consumer end process these messages.
Message Processing: Once a consumer finishes processing a message, it sends an acknowledgment to the queue to confirm successful processing. The message status changes from pending to processed. If the acknowledgment is not received within a certain time frame, the message is considered not processed and is made available to other consumers.
Handling Failures: If processing fails after a specified number of retries, the message is either stopped from being sent to consumers or moved to a Dead Letter Queue (DLQ) for further analysis.

Overall

The operation of AWS SQS is straightforward. When an application or system needs to send a message, it places the message into an SQS queue. The message remains in the queue until another process, known as a "consumer," retrieves it for processing.

For instance, if you are developing a bulk email sending application, you can utilize SQS to manage email sending requests. Each request is added to the SQS queue, and a separate process retrieves and processes these requests to send the emails.

With SQS:

SQS provides an API endpoint for producers to send messages and for consumers to read and process them. Messages sent to SQS can be in any format, including strings, XML, or JSON.
There is no limit to the number of messages that can be sent or processed by SQS.
Example Message Structure:

Here’s what a message in SQS might look like:

{
  "message_id": "a123d234g-82451-4qwf-a2f36-ee2135as25td",
  "receipt_handle": "AQEBLVo6YHmeJXe2np4Dz7Yf0UpRsIoM5Nk9T7W1jC/LuPp9ZG9b8QJv6EZo8L5XbKQ1dIn4yqV9G0YkCwH3j0TNRhM1gIb5LBcL2R1HrAo8ZLkH1lBG9ZPXTpG2bCu7HjpQK/lQ5Z8r1kLg+V8dV0b9mJ4CpF4Pne/Ay7GQfW9r8ZlF2r1uJvlzFv8nN3yV6d9LPokjImRTBxF8gP4C2QDLX0wF8NHdjpt7Ud9lKM0W5Q3be0Nsn9Nzj94QG+Fx7bT7SgA0lMLUyzk7OPg0xrP9kT+Rr1YmB9gSxfNl4+xk19vY5k/pS7Brz0YsUk0Tc8MtXpE61wXlsz6+VbOk2e8M6+HVtX39FNbGfhf5jHDu/k14EYtQv/g9ZRVUyQh6F9bC5Wv/g1Cs1VvoTLKq56yJQF9y3oRtx0OrX2bH2JlmBfs0aeY5rlQ==",
  "md5_of_body": "c923g2gf23g2g95asdg23sd12tg2g",
  "body": "8",
  "attributes": {
    "SenderId": "12345",
    "ApproximateFirstReceiveTimestamp": "13458591097096",
    "ApproximateReceiveCount": "1",
    "SentTimestamp": "1758926812155"
  },
  "md5_of_message_attributes": null,
  "message_attributes": {
    "EventType":"Updated",
    "bodyMessage":{
      "samplesA":"A",
      "sampleB":"B",
    }
  }
}

...

PostgreSQL & PgAdmin using Docker + Postgres Commands

Gavin Dang — Mon, 01 Apr 2024 03:08:50 +0000

Ref: https://migueldoctor.medium.com/how-to-run-postgresql-pgadmin-in-3-steps-using-docker-d6fe06e47ca1

Step1:
Download postgres docker image

$ docker pull postgres:latest

Step2:
Download pgadmin4

$ docker pull dpage/pgadmin4:latest

$ docker run -p 5050:80 -e 'PGADMIN_DEFAULT_EMAIL=pgadmin4@pgadmin.org' -e 'PGADMIN_DEFAULT_PASSWORD=admin' -d --name pgadmin4 dpage/pgadmin4

Step3:
Setup pgadmin 4
Open pgAdmin4 in your browser at: http://localhost:5050

Username: pgadmin4@pgadmin.org
Password: admin
Click Add New Server and fill in the following fields:

Server Name: pg (or whatever you prefer)
On the Connection tab:
Host: host.docker.internal
Port: 49153
Username: postgres
Password: postgrespw

Postgres Commands

sudo service postgresql start
sudo -u postgres psql

list of relations
postgres=# \dt

list of databases
postgres=# \l

Connect to specific database
postgres=# \c "database name"

List info in a table
databaseName=# \d user

show data in a table
TABLE "tableNamehere";

Docker Commands

Gavin Dang — Sat, 30 Mar 2024 07:26:33 +0000

Installing docker Ubuntu

sudo apt-get update
sudo apt install docker.io

sudo systemctl disable docker
sudo systemctl stop docker

sudo service docker start

Sample Dockerfile

FROM node:*-alpine

# update packages
RUN apk update

# create root application folder
WORKDIR /app

# copy configs to /app folder
COPY package*.json ./
COPY tsconfig.json ./
# copy source code to /app/src folder
COPY src /app/src

# check files list
RUN ls -a

RUN npm install
RUN npm run build

EXPOSE 7777

CMD [ "node", "./dist/main.js" ]

Docker commands

Images

Build docker image

docker build -t test-image-name .

Run image in interactive mode

docker run -it -p 7777:7777 test-image-name

Or run image in silent(daemon) mode

docker run -d -p 7777:7777 test-image-name

List all images

docker image ls

Remove all images at once

docker rmi $(docker images -q)

Containers

List all active containers

docker ps

List all active and dead containers

docker ps -a

Stop all running containers

docker stop $(docker ps -a -q)

Delete all stopped containers:

docker rm $(docker ps -a -q)

Other

Install help utils

apt-get install iputils-ping nmap

Jump into container shell

docker exec -it CONTAINER_ID /bin/sh