DEV Community: sngvfx

AWS: S3 Bucket access IAM policy example

sngvfx — Thu, 22 Dec 2022 06:43:54 +0000

Here is an easy way to grant all access to an S3 bucket and its objects to a specific user or group, you can use an S3 bucket policy similar to the following:

This policy grants all access (indicated by the "Action": "s3:" line) to the user or group identified by the "Principal" field (in this case, the user "demoteam" with the AWS account number **123456789012) for all objects in the bucket (indicated by the "Resource": "arn:aws:s3:::my-bucket/" line).

It's important to note that this policy grants the specified user or group full access to the bucket and its objects, which may not be appropriate in all cases.

Always customize the policy to grant less access to users and just based on your specific needs.

AWS: Lambda function

sngvfx — Fri, 16 Dec 2022 07:20:08 +0000

Amazon Lambda is a serverless computing platform that allows you to run code in response to specific events or triggers.

A Lambda function is a piece of code that is executed by the Lambda service in response to a trigger.

Here is an example of how you might use a Lambda function:

Imagine that you have a web application that allows users to upload images. When a user uploads an image (an event), you might want to automatically resize the image and create a thumbnail version. To do this, you could create a Lambda function that is triggered by the upload event and performs the image resizing.

The Lambda function might look something like this:

This is just one example of how you might use a Lambda function. You can use Lambda functions to perform a wide variety of tasks, such as processing data streams, automating back-end tasks, and integrating with other AWS services.

Apache Airflow - Steps by steps

sngvfx — Fri, 16 Dec 2022 06:00:33 +0000

Apache Airflow is an open-source platform for managing and scheduling data pipelines. It is commonly used in data engineering and data science to orchestrate and automate complex workflows, such as data ingestion, data transformation, and data analysis.

Here are the steps to build an Apache Airflow pipeline:

Install Apache Airflow: To use Apache Airflow, you need to install it first. You can install Apache Airflow either using pip or by downloading the source code and installing it manually. 
Set up a database: Apache Airflow uses a database to store its metadata, such as the list of tasks, their dependencies, and their execution history. You can use either a built-in database (such as SQLite) or a separate database service (such as MySQL or PostgreSQL). 
Define the pipeline: A pipeline in Apache Airflow is a directed acyclic graph (DAG) of tasks that need to be executed. To define a pipeline, you need to create a Python script and define a DAG object in it. Within the DAG, you can define individual tasks and their dependencies.

Create tasks: Tasks in Apache Airflow are represented by Operators, which are classes that define the behavior of a task. There are various types of operators available in Apache Airflow, such as BashOperator for executing a Bash command, PythonOperator for executing a Python function, and others. 
Set up the execution schedule: To run a pipeline, you need to specify when and how often the pipeline should be executed. This can be done by setting the start_date and schedule_interval parameters of the DAG object.

Test the pipeline: Before deploying the pipeline, it is a good idea to test it locally to make sure it is working as expected. You can do this by starting the Apache Airflow web server and running the pipeline in the web UI. 
Deploy the pipeline: Once you have tested the pipeline and are satisfied with the results, you can deploy it to the Apache Airflow environment. This can be done by committing the DAG Python script to a version control system and pushing it to the Apache Airflow environment. 
Monitor and maintain the pipeline: After deploying the pipeline, you need to monitor it to make sure it is running as expected. You can use the Apache Airflow web UI or the command-line interface to view the status of the pipeline and troubleshoot any issues that may arise. You should also regularly maintain the pipeline by updating it as needed and cleaning up old data to keep it running efficiently.

AWS: Difference between a Network Load Balancer and an Application Load Balancer

sngvfx — Fri, 16 Dec 2022 05:42:38 +0000

Today we will talk about the difference between a Network Load Balancer and an Application Load Balancer.

In Amazon Web Services (AWS), an Application Load Balancer (ALB) and a Network Load Balancer (NLB) are both types of load balancers that distribute incoming traffic across multiple targets, such as EC2 instances, in one or more Availability Zones. However, they differ in terms of the traffic they are designed to handle and the level of control they provide over the traffic routing.

An Application Load Balancer is a load balancing service that operates at the application layer (layer 7 in the OSI model). It is intended for load balancing of HTTP and HTTPS traffic and provides a number of features that enable you to route traffic based on advanced traffic routing rules and content-based routing rules.

In contrast, a Network Load Balancer is a load balancing service that operates at the connection level (layer 4 in the OSI model). It is designed for high performance and can handle millions of requests per second with very low latencies. It is best suited for load balancing of TCP traffic, and can also handle traffic for other protocols such as TLS and UDP.

In summary, an ALB is best suited for load balancing of HTTP and HTTPS traffic and provides a more feature-rich experience, while an NLB is better suited for load balancing of TCP traffic and is designed for high performance and low latencies.

Simple KAFKA streaming Pipeline

sngvfx — Thu, 15 Dec 2022 20:52:36 +0000

Let's say you are looking to explore the relationship between extreme weather and people's conversations about it on Twitter, you can gather and analyze weather and Twitter event data streams.

You will need the following (both in JSON format):

a Weather API to obtain real-time & forecasted weather data,
a Twitter API to also get real-time tweets & mentions.

The National Weather Service API can be accessed here

The Twitter API can be accessed here

Create a weather topic and Twitter topic in a Kafka cluster with some partitions and replications.
Create a weather producer and Twitter producer to publish the JSON data to the two topics, respectively.
Create a weather consumer and Twitter consumer to read the events from the two topics, and the bytes stored in the Kafka topics into event JSON data.
Use a DB writer to parse the JSON files and create database records, which can be written into the database with SQL insert statements.

Finally, you can query and visualize the database records in a dashboard to complete the end-to-end pipeline.

Additional infos:

What is a Kafka DB Writer:
A Kafka DB writer is a type of software that enables a user to write data from a database into a Kafka topic. It can be used for data integration, stream processing, or data ingestion. Kafka DB writers are often used for large-scale distributed systems, such as streaming analytics and real-time decision making.

KAFKA TOOL - ksqlDB:
KSQLDB is an open-source, cloud-native streaming SQL database built on Apache Kafka. It enables you to easily develop streaming applications that process data in real-time and respond to events as they occur. KSQLDB supports a wide range of operations including stream processing, time-based event-driven processing, windowed operations, joins, and more. It also provides a high-performance SQL query engine for accessing and analyzing data from Apache Kafka and other sources. With KSQLDB, developers can quickly build and deploy powerful streaming applications with a few lines of code.

TIC TAC TOE game in JS

sngvfx — Sat, 22 Oct 2022 21:03:03 +0000

This game is simple. Just place the mouse-over a square to play your turn. It will randomly play an X or O.
The first to match 3 lined-up X or O wins the game!

Simple Json Database

sngvfx — Sat, 22 Oct 2022 20:49:51 +0000

ZooDB - a simple database using CSS, JSON and HTML to display a pre-entered list of animals with details.

Click on the link below:

https://github.com/sngvfx/ZooDB

Simple Password Generator

sngvfx — Sat, 22 Oct 2022 16:30:16 +0000

Here is a simple python generator that use the "Random" module with a random.sample() method. The generated password is a combination of upper and lowercase letters, numbers and symbols. To make the password more complex and not easy to hack, the length of the password has been set to 20.

Give it a try, feel free to improve it and let me know!

WELCOME TO SNGVFX

sngvfx — Fri, 21 Oct 2022 19:36:12 +0000

Welcome to my coding page. Here you will find a list of my current and past coding projects.... Created using a combination of the following programming languages: Python, Javascript, CSS, PHP, Linux/Bash and Database MySQL, PostgreSQL and MongoDB.

My projects include:

Simple Password Manager
Pro Password Manager V.1
FolderDeletor V.1
Video Processor V.2
Serato Live Track Reader V.1
Simple Music File & Folder Organizer V.1

and more...
I will be sharing some here and on GITHUB.

ETL Process - The ABC of the DATA Engineer

sngvfx — Sun, 06 Feb 2022 15:24:41 +0000

ETL - the process of extracting, transforming and loading data, also called streaming data process, is the foundation of data engineering.

As someone who is switch career (From Web development/Front-End), learning all the tools, processes and concept behind ....