DEV Community: Amzar

How to Re-tag an Image in AWS ECR

Amzar — Fri, 19 Jul 2024 20:18:32 +0000

Are you managing Docker images in AWS ECR? Sometimes, you need to retag them to keep things organized. In this quick guide, we'll show you how to retag your Docker images in AWS ECR Let'sy. Let’s get started!

Let's consider an image named hello-world-program with the following tags.

current tag	new tag
[untagged]	0.0.2

Steps

Setup AWS CLI
Navigate to AWS ECR and choose the images you want to update. Copy the image digest (e.g., sha256:xxx), which is a unique and unchangeable identifier. Each image's content has its own distinct digest.
Open terminal

Run the following command to retrieve the image manifest and store it in the MANIFEST variable:

MANIFEST=$(aws ecr batch-get-image --repository-name {REPO_NAME} --image-ids imageDigest={IMAGE_DIGEST} --output json | jq --raw-output --join-output '.images[0].imageManifest')

Run the following command to update the image:

aws ecr put-image --repository-name {REPO_NAME} --image-tag {NEW_TAG} --image-manifest "$MANIFEST"

Replace {REPO_NAME} with the name of your repository, {IMAGE_DIGEST} with the actual digest you copied from AWS ECR, and {NEW_TAG} with your desired tag

Refresh the page, and you’ll see that the existing [untagged] image is now updated to 0.0.2.

If you're on macOS and encounter the issue jq: command not found, check this guide for help.

Reference

Guide to Validating Sign-In with Apple Tokens in Python

Amzar — Sat, 23 Dec 2023 16:19:25 +0000

Background

The integration of Sign In with Apple into applications has become increasingly popular due to its convenience and privacy-focused approach. However, navigating the token validation process and retrieving user information can be a complex task for developers. This article aims to demystify this process specifically for Python developers, offering a step-by-step guide on how to validate tokens obtained through Sign In with Apple and efficiently access user data within your application. By delving into the intricacies of token validation and user information retrieval, this comprehensive guide aims to equip developers with the knowledge and tools necessary to seamlessly implement and utilize Sign In with Apple functionality in their Python-based applications.

Prerequisite

This article is not focussing on how to get the value from Apple Developer page.

Client ID (client_id)
Client Secret (client_secret) - A JSON Web Token (JWT) generated by the developer

Generate client secret

Generate a signed token to identify your client application. [2]

key_id = None
team_id = None
client_id = None
private_key = None  # .p8 key file

headers = {"kid": key_id}
open_private_key = open(private_key)
private_key_data = open_private_key.read()


payload = {
    "iss": team_id,
    "iat": datetime.now(),
    "exp": datetime.now() + timedelta(days=180),
    "aud": "https://appleid.apple.com",
    "sub": client_id,
}

client_secret = jwt.encode(
    payload, private_key_data, algorithm="ES256", headers=headers
).decode("utf-8")

Generate and validate tokens

Validate an authorization grant code originating from your frontend interface.

headers = {"content-type": "application/x-www-form-urlencoded"}

data = {
    "client_id": client_id,
    "client_secret": client_secret,
    "code": id_token,
    "grant_type": "authorization_code",
}

res = requests.post(URL, data=data, headers=headers)
values = res.json()

Below is the response you'll get after success make request

{
  "access_token": "adg61...67Or9",
  "token_type": "Bearer",
  "expires_in": 3600,
  "refresh_token": "rca7...lABoQ",
  "id_token": "eyJra...96sZg"
}

access_token: token from frontend
token_type: type for OAuth 2.0 framework
expires_in: lifespan or validity duration of an access token
refresh_token: renews access without re-login
id_token: token to get the user details (next action)

Get the user details

decoded = jwt.decode(
                id_token,
                "",
                verify=False,
                options={"verify_signature": False},
                algorithms=["HS256"],
            )

References

Basic Kafka Stream in Python

Amzar — Sat, 11 Mar 2023 19:25:02 +0000

Introduction

Kafka is an open-source distributed streaming platform developed by the Apache Software Foundation. It is designed to handle large volumes of real-time data streams and is used for building real-time data pipelines and streaming applications.

For instance, if you would like to have a real-time event log fetch from the source (Twitter, etc) and insert it into the target (CSV, SQLite, etc), you can leverage the Kafka library in Python. So, you need to know what are producer and consumer in Kafka.

Producer and Consumer

Producer is a program or application that sends data or messages to a Kafka topic.
Consumer is a program or application that reads data or messages from a Kafka topic.

How to setup Kafka in Python?

Now, let's start with requirements to configure Kafka in Python.

Requirements

Kafka server

You can start with local installation or AWS and GCP for advanced method). In this tutorial, I am launching Kafka server using Docker Compose.

Assume Docker is installed and running or run docker --version to confirm

Create a docker compose file (docker-compose.yaml) and copy below snippet.

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Then, we could build the Docker
docker-compose up --build --force-recreate -d

To confirm the containers are up, we need to check the status.

Docker Desktop

Command

╰─➤  docker-compose ps
      Name                   Command            State                      Ports
---------------------------------------------------------------------------------------------------
kafka_kafka_1       /etc/confluent/docker/run   Up      0.0.0.0:29092->29092/tcp, 9092/tcp
kafka_zookeeper_1   /etc/confluent/docker/run   Up      0.0.0.0:22181->2181/tcp, 2888/tcp, 3888/tcp

Okay, now it shows the container is successfully built and running. Next step, how to create a topic and send the message?

Kafka producer (Python)

Create a topic

from confluent_kafka.admin import AdminClient, NewTopic

admin_client = AdminClient({"bootstrap.servers": "localhost:29092"})

topic_name = "my_first_topic"
num_partitions = 3

new_topic = NewTopic(topic_name, num_partitions)

try:
    admin_client.create_topics([new_topic])
    print(f"{topic_name} is created!!")
except:
    print(f"{topic_name} is not created!!")

I am using confluent_kafka to integrate with Kafka server.

Also, we can define the number of partitions for the topic. Important to know that number of partitions cannot be changed after the topic has been created.

If you have a topic with 10 partitions, you can have up to 10 consumers processing messages from that topic in parallel, each handling messages from its assigned partition.

╰─➤  python kafka_topic.py
my_first_topic is created!!

Okay, the topic has been created!

Note: If you first forgot to create a topic, no worries. It should create when you post the message to the new topic.

Post a message

import json
import uuid
from confluent_kafka import Producer


def post_message(producer, data):
    try:
        producer.produce("my_first_topic", json.dumps(data).encode("utf-8"))
        producer.flush()
        print(f"message posted!! --> {data['comment']}")
    except:
        print("failed to post message")


data = {
    "user_session_id": str(uuid.uuid4()),
    "user_name": "John Doe",
    "comment": "Malaysia Boleh!",
}

producer = Producer({"bootstrap.servers": "localhost:29092"})
post_message(producer, data)

When you run the Python script, it could post the message

╰─➤  python producer.py
message posted!! --> Malaysia Boleh!

Kafka consumer (Python)

Okay, let's move to the consumer to check whether the data is really posted.

from confluent_kafka import Consumer

consumer = Consumer(
    {
        "bootstrap.servers": "localhost:29092",
        "group.id": "python-consumer",
        "auto.offset.reset": "earliest",
    }
)

consumer.subscribe(["my_first_topic"])

while True:
    msg = consumer.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        print("Error: {}".format(msg.error()))
        continue
    data = msg.value().decode("utf-8")
    topic = msg.topic()
    ts = msg.timestamp()
    print(data, topic, ts)

And, here is the output when you run the script

╰─➤  python consumer.py
{"user_session_id": "5321202b-694a-4d82-9d2b-97174303595e", "user_name": "John Doe", "comment": "Malaysia Boleh!"} my_first_topic (1, 1678561082329)

Here is the demo when you post a message, the consumer can print it out within a second.

Enjoy ~

What Python libraries should I learn? [2023 UPDATED]

Amzar — Sun, 05 Feb 2023 05:01:03 +0000

Python is one of the most popular programming languages mentioned by Coursera. As you know, Python can do anything from development to statistical analyses, and also, many standard libraries are provided.

Popular Python libraries for Starter

Here is my recommendation of Python libraries that starter should know.

1. os

Let say you want to interact with operating system by code, os help a lot. You can list of the directory and also rename, delete or anything by the code.

import os

os.getcwd() # get current directory
os.listdir() # list current directory
os.listdir('/Users/mohamadamzar/code') # list specific directory
os.rename(old_filename, new_filename) # rename file

2. requests

You can use requests to interact with the API. It provide REST API method such as GET, POST, PUT and DELETE. If you are into the development, must know how to use the library.

3. BeautifulSoup

You can do data extraction (HTML and XML) easily by using BeautifulSoup. It parses the data with beautiful formatting; you can extract it by tag and div. You don't need to worry about the complicated tag; it can be solved using the library. Javascript? It will only be workable with simple Javascript 😉

4. cv2/opencv

Interesting library to try. It can interact with the image; read and display the image, turn it to greyscale, save the image from the link, create a circle in the image, and many more.

import cv2
img = cv2.imread('range-rover.png', cv2.IMREAD_COLOR)

5. pandas

Pandas is a library to interact with multiple data sources. You not only can read the data, but you can analyze and manipulate the data. For the data analyst, it can be used for data wrangling and cleaning. Also, you can explore the simple ETL using pandas.

There's a lot more to explore; I will update the post from time to time!

Keep your Windows always on!

Amzar — Tue, 31 Jan 2023 18:00:05 +0000

When you're working on something else that needs away from the keyboard, your Windows always automatically locks due to the setting (because you don't have admin access to change the setting due to company restrictions).

Usually, people will open YouTube repeatedly, which is tiring since YouTube also has a policy that if you're idle, it will auto-pause.

Solution

With Python, you can code a script to press the key for a certain time range.

...
def main(args):
    try:
        print(progname)
        console.print(
            "\nRunning with [bold cyan]{}[/bold cyan] key.".format(args.button)
        )
        no_lock(args.button)

    except KeyboardInterrupt:
        print("\n[Process Stopped]", flush=True)

    except Exception as ex:
        print({"main": ex})
...

Only works on Windows :)

Full Code

Github follow the installation step in README.md

I build my own Shopee discount alert on Slack channel!

Amzar — Tue, 31 Jan 2023 17:32:13 +0000

I love discounts and I bet you are too. My fav online shopping platform is Shopee MY. But, how come I can always 24/7 checking on the App to get notified when the seller promotes the products with DISCOUNT? It is not works for me.

So, I build my own Shopee alert.

How to get the information?

The first thing is I need to get the information from the Shopee. But, do I have the Shopee API? No. I don't. My next option is web scraping. By using requests and BeautifulSoup, it is possible to do so with a few tricks.

...
headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0",
    "X-Requested-With": "XMLHttpRequest",
    "Referer": link,
}
page = requests.get(URL, headers=headers)

if page.status_code == 200:
    soup = BeautifulSoup(page.content, "html.parser")
...

Where do I store the information?

Next, I must store the data to compare the price/discount before sending the alert. Why do we always need the alert when the price is not the lowest?

I build for my own use; I just need a simple database. Below is the database in my list:

Google Sheet
Baserow ✅
Excel and store in AWS S3

After few consideration, I choose to use Baserow. But why?

open-source with API capability
backend using Postgres
better admin panel

...
res = requests.get(
    url=f"{base_url}database/rows/table/{table_id}/?user_field_names=true",
    headers=headers,
)

if res.status_code == 200:
    data = res.json()
    res_data = data["results"]
    df = pd.DataFrame(res_data)
...

Notification or alert

If the price is the lowest (compared with the historical data), it should alert me.

For this project, I am using Slack channel to get the alert. Slack is one of the easiest alert integration compared to Telegram or Discord.

...
def sendToSlack(itemRes, no_data=False):
    if no_data:
        payload = getPayloadEmpty()
    else:
        payload = getPayload(itemRes)

    slack_webhook = os.environ.get("SLACK_URL")
    requests.post(slack_webhook, json=payload)
...

Now, let's see how the alert looks like

It's contain the discount and link to the product.

Deploy the script

I am using cron job running in the AWS Lightsail instance. Schedule to run the bash script every 3rd hour from 1 AM to 11 PM.

export PYTHONPATH=/home/amzar/Documents/python_env/shopee-tracker/bin/python3 

/home/amzar/Documents/python_env/shopee-tracker/bin/python3 

/home/amzar/Documents/shopee-tracker/app.py

Cron expression 0 1-23/3 * * * /home/amzar/scripts/run_shopee_tracker.sh

Full Code

Github follow the installation step in README.md

requests.get (python)

Amzar — Tue, 31 Jan 2023 16:55:06 +0000

get is the method to retrieve information from the API. You can choose the result in JSON or raw text format.