DEV Community: Kairat

Beyond LIKE: Advanced Text Search and Keyword Matching in Postgres using Full Text Search

Kairat — Sat, 23 Nov 2024 00:32:38 +0000

Imagine you’re collecting video data from YouTube using their API and storing it in a Postgres database. Now, you want to automatically tag certain videos based on specific keywords stored in another table or enable text search functionality.

The simplest solution would be to create a database view using the basic LIKE operator. You could then set up a cron job to periodically fetch new records and assign tags to them.

This approach might work for simple use cases, but it has several limitations:

while LIKE supports basic substring matching, it has significant limitations. You could use ILIKE for case-insensitive searches or ~ for regex matches, but these are still quite basic. Achieving advanced features like ranking, synonym recognition, or phrase-based matching is either extremely difficult or impossible with these tools.
as your data grows, performance becomes a concern. The engine has to scan through the text to find matches, which can be computationally expensive. Indexing can improve performance to some extent. For example, a B-tree index works well for patterns like "rust%" (starts with). However, searches such as "%rust%" (anywhere in the text) or "%rust" (ends with) cannot leverage the index effectively and will fall back to full-text sequential scans.

Because of that, using LIKE and similar operators might not be the most suitable solution for this use case.

Fortunately, Postgres has built-in full-text search functions that address these limitations.

To demonstrate how it works, let's create simple youtube_videos table.

CREATE TABLE IF NOT EXISTS
  "public"."youtube_videos" (
    "id" uuid DEFAULT "gen_random_uuid" () NOT NULL PRIMARY KEY,
    "title" text
  );

It has title column and we gonna run our searches against it.
But to be more specific, tokenized version of it.

And you might be wondering what do I mean by that?

Well, Postgres has this function - to_tsvector() which stands for "to text search vector". Using this function we can transform our video title into searchable tokens that can be used for comparisons.

This conversion involves lemmatization which normalizes word. It involves making word lower-cased, removing suffixes (such as es in Eng) and stop words (such as a and the in Eng). This step is done by the use of dictionaries. Various standard dictionaries are provided, and custom ones can be created for specific needs.

To avoid making this transformation every time we need to search for something, let's store it in the column of type tsvector.

alter table youtube_videos
add column fts_title tsvector generated always as (to_tsvector('english', title)) stored;

Running this query will create a new column in our videos table. It will be generated every time we update the title column (on INSERT and UPDATE).

First parameter passed to function is a dictionary name. In this example as you can see we used basic english dictionary. You can read more about Postgres dictionaries here.

Also, in this specific use case, we know for sure, every video has a title, but if you are dealing with nullable columns, it's better to use coalesce(title, '') to handle null values.

To speed up the text search process even more, let's create GIN index on column. As this column is of type tsvector, regular B-Tree index won't work in this case.

create index title_fts_inx on youtube_videos using gin (fts_title);

Now, after inserting a few records into the table, we can take a look at what fts_title value looks like:

As you can see, normalization was applied to the title: ""

"stocks -> stock"
"to, you, in" were dropped

And now, to actually find keyword matching records, we need to use the to_tsquery() function that converts a query string into tokens to match.

select
  *
from
  youtube_videos
where
  fts_title @@ to_tsquery('english', 'rust');

Here, @@ symbol is the "match" symbol for full-text search.
Execution time for this command is roughly between 0.05 - 0.065 ms.

For comparison, running search using ILIKE operator on indexed title column, is way slower (1.45 ms - 1.50 ms).

create index title_inx on youtube_videos(title)

SELECT
  *
FROM
  youtube_videos
WHERE
  title ilike '%rust%';

Or let's say we wanna get all videos about stocks (maybe I wanna buy PLTR or Tesla).

Running query like that:

SELECT
  *
FROM
  youtube_videos
WHERE
  title ilike '%stock%';

Returns these 2 records:

Maybe we can use a bit different query:

SELECT
  *
FROM
  youtube_videos
WHERE
  title ~* '\ystock\y';

Returns none as stock is singular.

But running this query, returns the desired result.

SELECT
  *
FROM
  youtube_videos
WHERE
  title ~* '\ystock\y';

And let's say we have keywords table.

Now we can create a view that has all the videos containing any of those keywords:

CREATE OR REPLACE VIEW videos_with_keywords AS
select
  yv.*
from
  youtube_videos yv
  join keywords on (yv.fts_title @@ to_tsquery(keywords.keyword))

We can also do partial searches:

select * from youtube_videos where fts_title @@ to_tsquery('english', 'sto:*')

While this article focused on the basics of full-text search, PostgreSQL offers many advanced features that you can explore further. Here are some great resources to dive deeper:

Docs
Supabase Article

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

Safe Content, Happy Users: Azure AI Content Safety

Kairat — Tue, 27 Feb 2024 22:06:00 +0000

The Internet is arguably one of the most important inventions in human history. However, since its creation, it has also become a source of concern. Not only is there an abundance of misinformation, but the prevalence of harmful and offensive content has turned it into a daunting place. This issue has reached alarming levels, leading to severe consequences in some cases.

As a result, major organizations such as Twitch, YouTube, Facebook, and others that rely on user-generated content have been actively working on implementing measures to filter out offensive content. The underlying structures of these platforms are complex and just a few years ago, creating an effective profanity-filtering tool would have been a difficult task.

Fortunately, with the advent of AI tools such as Gemini and ChatGPT, as well as pre-trained moderation language models (LLMs), integrating content filtering into your application has become much more feasible.

In this article, I want to explore the Azure AI Content Safety service and demonstrate how it can be used to detect potentially harmful content.

Whether you are trying to build an app where people can leave comments under articles or aiming to develop a new Instagram to compete with Mister Zuckerberg, you would need to manage user-generated content - uploaded images, comments, etc. And that's where you might benefit from AI Safety Tool - perhaps you want to block people from uploading offensive or harmful content or at least be notified that potentially unsafe content was uploaded.

This tool is one of Azure's AI Services and it precisely addresses the need discussed above.

From docs:

AI Content Safety detects harmful user-generated and AI-generated content in applications and services. Azure AI Content Safety includes text and image APIs that allow you to detect material that is harmful.

It offers 4 types of analysis:

text analysis
image analysis
jailbreak risk detection (scans text for the risk of a jailbreak attack on a Large Language Model)
protected material text detection (scans AI-generated text for known text content (for example, song lyrics, articles, recipes, selected web content))

Today, we're only going to focus on the first aspect - text analysis and how you can integrate it into your application.

But before going into the details of the actual implementation, I'd like to mention one more thing - the interactive Content Safety Studio.
This platform enables users to view, explore, and experiment with sample code for detecting harmful content across various modalities. Additionally, it empowers you as a developer to monitor key performance indicators (KPIs) such as technical metrics (latency, accuracy, recall) and business metrics (block rate, block volume, category proportions, language proportions, and more).

To begin the actual implementation, you'll need to have an Azure account with an active Azure Subscription.

To create a new instance of the AI Content Safety service, navigate to the Azure home page and start typing "content safety" in the search bar. This will bring up relevant services related to content safety.

Let's create a new instance.

Choose the resource group, name, location and pricing tier.

You can also configure networking and identity settings if needed. However, if you prefer, you can leave everything by default for simplicity.

Now that everything is set up, go ahead and click on the "Create" button to proceed with creating the instance.

Navigate to the newly created resource, then select "Keys and Endpoint" from the menu. This section will provide you with the necessary authentication keys and endpoints to interact with the AI Content Safety service.

If you go to the documentation, you will find a lot of ways to use Content Safety - you can call it directly through REST API or use the language-specific packages (JS, Python, C#).

I am going to use REST - that's why we needed that endpoint and keys.

For demonstration purposes, I've created a simple Astro page where users can leave comments. These comments will be validated using the Content Safety service across four harm categories: Self-Harm, Sexual, Violence, and Hate & Fairness.

You can read more about those here.
For each category, we will also receive the severity level rating, which indicates the severity of the consequences of displaying the flagged content.

In the case of text analysis, it supports a full 0-7 severity scale, where 0 represents content deemed safe and 7 signifies a high severity level.

You don't have to concern yourself with HTML structure and CSS, as we won't be focusing on them in this article. The only code we care about is the script tag where we make a call to the Content Safety endpoint.



<script>
    interface ContentSafetyResponse {
        blocklistsMatch: {blocklistItemText: string; blocklistItemId: string;blocklistName: string}[];
        categoriesAnalysis: { category: string; severity: number }[];
    }

    const comment : HTMLTextAreaElement = document.getElementById("comment") as HTMLTextAreaElement;
    const submitBtn = document.getElementById("submitBtn");

    submitBtn?.addEventListener("click",async ()=> {
        if (comment) {
            const data = await fetch('your-endpoint', {
                method: "POST",
                headers: {
                    "Content-Type": "application/json",
                    "Ocp-Apim-Subscription-Key": "key"
                },
                body: JSON.stringify({
                    "text": comment.value,
                    "categories": [
                        "Hate", "Sexual", "SelfHarm", "Violence"
                    ],
                    "blocklistNames": [
                        "SwearWords"
                    ],
                    "outputType": "FourSeverityLevels"
                })
            })
            const json: ContentSafetyResponse = await data.json()
            json.blocklistsMatch.map((match)=> {
                const blocklist = document.getElementById("blocklist");
                if (blocklist) {
                    blocklist.innerText = blocklist.innerText + " " + match.blocklistItemText
                }
            })
            json.categoriesAnalysis.map((category)=> {
                if (category.category.toLowerCase() == "hate") {
                    const hateLevel = document.getElementById("hateLevel");
                    if (hateLevel) {
                        hateLevel.innerText = category.severity.toString()
                    }
                }
                if (category.category.toLowerCase() == "violence") {
                    const violenceLevel = document.getElementById("violenceLevel");
                    if (violenceLevel) {
                        violenceLevel.innerText = category.severity.toString()
                    }
                }
                if (category.category.toLowerCase() == "sexual") {
                    const sexualLevel = document.getElementById("sexualLevel");
                    if (sexualLevel) {
                        sexualLevel.innerText = category.severity.toString()
                    }
                }
                if (category.category.toLowerCase() == "selfharm") {
                    const selfharmLevel = document.getElementById("selfharmLevel");
                    if (selfharmLevel) {
                        selfharmLevel.innerText = category.severity.toString()
                    }
                }
            })
        }   
    })
</script>

We make a POST request to the endpoint, we'll assign the value "application/json" to the "Content-Type" header and provide "Ocp-Apim-Subscription-Key" as another header.
The request body should contain the following parameters:

text - content that needs to be validated
categories - all 4 or some of them
outputType - can be FourSeverityLevels (possible values: 0,2,4,6) or EightSeverityLevels (possible values: 0,1,2,3,4,5,6,7)
blocklistNames - names of custom blocklists (we will discuss this in more detail later on)
haltOnBlocklistHit - when set to true, further analyses of harmful content won't be performed in cases where blocklists are hit

Let's test it out.

As you can see, severity levels for all 4 categories are rated 0 which means content is safe.

Now let's try something different (only an example, you can do it only in GTA RP)

As you can see here, we changed the context on how we want to use the axe and now we are getting a violence level of 4. It's not just a keyword matching, but it does understand the context as well.

You may have already observed the "blockListsMatch" property in the response.

What is that?

So, for most cases default AI classifiers are good enough, but sometimes you might need more than just the default ones.
This is where blocklists come into play. Essentially, a blocklist is a curated list of words or phrases that, if detected in the text, will trigger a failure in the content check.

As an example, the word "hoser" is a Canadian playful insult (like a loser). Let's say you do not want any insults on your platform.
If someone gonna use this word, with the default classifiers, it will pass the check (in comparison with the word "loser")

How to fix that?
You already know the answer - blocklists.
One way to create a blocklist is through API. We not gonna do that instead let's use Safety Studio.

Here you can also create a new blocklist.

I am not creating a new one as I already created one before, so let's just use it. It's called SwearWords - the name is important as we gonna use it as one of the body parameters.

Now let's add the word "hoser" to that list.

If you were paying attention, in our code above, we specified blocklist names - the "SwearWords" list was there.



"blocklistNames": ["SwearWords"]

Now, let's try to validate the last sentence one more time and see the results.

Did something change? Severity levels are still at 0. But look at the second property in the response object - blocklistsMatch. It is no longer empty - instead, it shows us that our text contained the word from the SwearWords list and it should fail the validation.

That was just a sample of how you can use it in your apps and there is so much more you can do with this tool. This article aims to introduce you to the capabilities of the AI Content Safety service and demonstrate how you can leverage it to enhance your projects.
It's worth noting that the AI Content Safety service has undergone extensive testing and is not merely a rough prototype in the development phase. It has already been successfully integrated into various Microsoft products, including Copilot, showcasing its reliability and effectiveness.

There is a great video on its functionality with Sarah Bird who was leading the project. Check it out!

Here is the API documentation.

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

JavaScript Efficiency War: Astro.js vs Next.js

Kairat — Wed, 21 Feb 2024 21:51:14 +0000

Astro has been on the market for a while but it's still considered a relatively new library. In 2023 there were a lot of talks about it and I decided to give it a try.

Before I was primarily using React and Next 13 but most of the applications I was working on were informative simple websites with mostly static data. That is why Astro seemed very attractive as it is the perfect library for such use cases.

Like most people who tried Astro, I liked it and in this article, I want to particularly discuss how Astro impacts the performance of your website.

To start, I want to talk a bit about the new architectural pattern that Astro is trying to popularize. It's called island architecture and Astro docs have a separate page explaining this concept.
Here I just want to briefly highlight the main ideas behind it so you have more context on why Astro is so good at performance.

If you've never heard of this technique (architectural pattern), there is a chance you might know it by a different name - partial or selective hydration.

The main idea behind it is relatively simple.
First, let's define what's an island.

Basically, an island is any interactive UI component on the page.

An island always runs in isolation from other islands on the page, and multiple islands can exist on a page.

So, now let's move on to non-interactive components on your pages.
All other UI components - non-interactive ones are static, lightweight, server-rendered HTML & CSS.

And now what we have in the end - static components rendered on the server along with dynamic ones - those that can be "hydrated" on the client.

What does this all mean?
React, Next.js and other SPAs use a monolithic pattern where the entire website is one large JavaScript application (not in the case of Next.js) and we end up shipping a lot of unused JavaScript to the browser which we don't even gonna use. With these libraries we can't selectively (partially) and strategically hydrate certain components, instead, we hydrate the entire app.

If all of it does not make sense to you, let me show an example of how it works.

Let's say we have a web page with the following structure.

If we would build the following static page using Next.js, we would render it on the server and send JavaScript to the client to hydrate dynamic components, right? The catch here is that we not only gonna send the JS required for Header & Carousel components. but also JS for static components as well and we don't need it cause we know those 3 components are static and there is no need for us to hydrate those on the client.

That's why we have Astro, and what would happen if we were to build this same page with it?
We strip all non-essential JavaScript from the page (it's done automatically) which slows down the website and instead, we only send JS needed for dynamic components' hydration.

I hope it all makes more sense now, but if you are still feeling a little bit confused, I highly recommend watching this video from Vite Conference by Nate Moore.

To give you even more visualization, I built this super simple web page where users can enter their email and after clicking the subscribe button confetti is shown.

I built it twice - using Astro.js and Next 13 static export (can be done by specifying output: 'export' in the next configuration file).

That's how the code looks if using Astro.js



<html lang="en">
    <head>
        <meta charset="utf-8" />
        <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
        <meta name="viewport" content="width=device-width" />
        <meta name="generator" content={Astro.generator} />
        <title>Astro</title>
    </head>
    <body>
        <div class="w-screen h-screen bg-blue-800 flex justify-center items-center">
            <div class="flex w-[60vw] h-[70vh] bg-neutral-300 rounded-3xl px-8 py-4">
                <div class="flex flex-col basis-1/2 gap-y-4">
                    <h1 class="text-4xl">Stay updated</h1>
                    <h2>Join 60+ product managers receiving monthly updates on</h2>
                    <ul class="list-disc list-inside">
                        <li>Product discovery and building what matters</li>
                        <li>Measuring to ensure updates and a success</li>
                        <li> And much more!</li>
                    </ul>
                    <h3 class="pt-8">Email Address</h3>
                    <form id="newsletterform" class="flex flex-col gap-y-2">
                        <input required type="email" class="w-[90%] p-4"/>
                        <button id="subscribeBtn" type="submit" class="w-[90%] p-2 bg-red-400 hover:bg-red-500">
                            Subscribe 
                        </button>
                    </form>
                </div>
                <div class="basis-1/2 rounded-3xl bg-pink-400 overflow-x-hidden">
                    <img src="TestImg.jpg" alt="Alternative Text for Image" class="h-full object-cover"/>
                </div>
            </div>
        </div>
    </body>
</html>

<script>
    import confetti from 'canvas-confetti';
    const form = document.getElementById('newsletterform');
    form?.addEventListener("submit", (e)=> {
        e.preventDefault()
        confetti()
    })
  </script>

Here is the Next 13 version:

page.tsx



import Form from "@/components/button";

export default function Home() {
  return (

    <div className="w-screen h-screen bg-blue-800 flex justify-center items-center">
      <div className="flex w-[60vw] h-[70vh] bg-neutral-300 rounded-3xl px-8 py-4">
        <div className="flex flex-col basis-1/2 gap-y-4">
          <h1 className="text-4xl">Stay updated</h1>
          <h2>Join 60+ product managers receiving monthly updates on</h2>
          <ul className="list-disc list-inside">
            <li>Product discovery and building what matters</li>
            <li>Measuring to ensure updates and a success</li>
            <li> And much more!</li>
          </ul>
          <h3 className="pt-8">Email Address</h3>
          <Form />
        </div>
        <div className="basis-1/2 rounded-3xl bg-pink-400 overflow-x-hidden">
          <img src="TestImg.jpg" alt="Alternative Text for Image" className="h-full object-cover" />
        </div>
      </div>
    </div>

  );
}

button.tsx



"use client"

import confetti from 'canvas-confetti';

export default function Form(): JSX.Element {
    return <form id="newsletterform" onSubmit={(e) => {
        e.preventDefault();
        confetti();
    }} className="flex flex-col gap-y-2">
        <input required type="email" className="w-[90%] p-4" />
        <button id="subscribeBtn" type="submit" className="w-[90%] p-2 bg-red-400 hover:bg-red-500">
            Subscribe
        </button>
    </form>;
}

I built it using the npm run build command and then started the application in production mode (npm run start for Next and npm run preview for Astro).

Now, we go to the network tab to see what requests we make to display this page. Particularly we are interested in JS requests.
And what do we see?

In Next.js, we have 5 requests

While in Astro, it's just one

In total, we made 5 requests in Astro (HTML document, 1 CSS file, one JS, 2 images (body and favicon) and in Next 9 - +4 JS files.

Astro

Next

If you want to see, what those JS files consist of, you can use bundler analyzer tools.

I used this one for Astro - vite-bundle-visualizer.

And for Next, there is actually a package built for it - @next/bundle-analyzer

As you can see from this image, Next.js comes with a bunch of default packages it needs for proper work - and we, as developers, could benefit from those a lot but in more complex apps with more features while simple static informative websites don't need those.

In terms of Lighthouse performance, Astro is better here as well (just a bit and of course it's gonna vary a lot based on multiple factors)

Astro

Next

I know it is a super simple web page and in reality, you would have much more complex structures but it still shows the difference between these 2 frameworks and how Astro improves performance by stripping non-essential JavaScript code.

In the end, just wanna say that I am gonna continue using Next 13 as well as Astro. The main point of this article was to show you that as developers we shouldn't rely on one tool and be flexible depending on the situation.

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

Frustrated with Tracking Grocery Costs? Find Your Answer Here

Kairat — Sun, 18 Feb 2024 20:31:01 +0000

How much do you spend on groceries per month?

Is it $500? Or maybe $600–700? For some, it might be more than $1k.

There are a lot of factors that affect this number — your gender (statistically males spend more than females), your diet (if you try to eat healthy, your bill will likely be higher), family size (those who have kids spend much more), area you live in, etc.

But probably all of us noticed that the cost of the average grocery bill went up a lot. It all makes sense — inflation was crazy in the past few years, constant talks about recession, etc. Of course, it impacted grocery prices as well.

But my question here is how do you keep track of your grocery expenses?

Do you use Excel, Google Sheets or any app that helps you with that? There are dozens of shopping list apps out there, budget apps like Mint, YNAB, etc.
But what is the functionality of these apps?

Shopping list apps simplify your shopping experience, Expense trackers show how much you spend on groceries overall, and coupon apps like Flipp & Reebee help to find the best deals & discounts at your stores.

All of that sounds great! But do you know how much you spend on a specific grocery item? What is the average price you pay for this product? Where does this item cost less? Did the price for that product go up or down?

None of these apps have this specific functionality.

Flipp is relatively close to what I want but it still aims different goal.

Wouldn’t it be nice to keep track of prices for the items you typically purchase, monitor changes over time, compare prices across different stores, and gain insights into your grocery shopping habits such as identifying the most expensive and cheapest items, the most and least purchased items, the average bill amount, etc?

There are multiple Reddit threads where people discussed the same thing:

App/program that automatically tracks which grocery store has the best price for an item?

Any apps to track my grocery purchases and prices

Receipt scanning & price tracking app for groceries?

Simple price tracking app for groceries?

Price book app for tracking grocery costs across several stores

Seems like most people use spreadsheets for this or even paper books!

So, wouldn’t it be great to automate this process and have an app?

Instead of constantly filling Excel sheets or writing in a paper book, wouldn’t it be much easier to just scan your paper receipt, and get all the necessary data from it such as item name, unit (kg, ea, etc.), price per unit, quantity, and total, save it and then have statistics on this particular purchase, specific items and their price fluctuations?

Using powerful OCR software (Optical Character Recognition), users would be able to scan receipts, and in most cases, the results would be quite accurate. The challenge with paper receipts lies in the quality of the paper they are printed on, which may not always be optimal. As a result, there are instances where OCR software may struggle to scan them accurately or may produce errors in the scan. However, users will always have the option to correct any inaccuracies manually or enter the data manually themselves.

After saving this data, users will always have access to their purchase history. They can view exactly how many purchases they made and how much they spent in total each month, over the last three months, or within any other custom date range.

Users will also have access to detailed statistics for each item they’ve ever purchased. This includes the average, lowest, and highest prices for the item, total expenditure on it, the number of times it was purchased, purchase dates, price fluctuations over time, and where the item costs less.

With this comprehensive data at their disposal, users can make informed, data-driven decisions. For example, they may decide it’s time to switch to another brand to save money, or perhaps it’s best to stop purchasing the item altogether.

I’ve been dedicating my free time to developing a rough prototype of this app. While I’m currently in the process of refactoring the backend, I’ve had the chance to use it myself. Despite its current limitations, it has already provided valuable insights into my grocery spending behaviour.

Using the prototype, I’ve gained a clearer understanding of my spending habits. It has highlighted areas where I was overspending on unnecessary items and revealed price fluctuations in some of my frequently purchased items. This insight made me consider switching to more cost-effective brands. Overall, the prototype has proven to be a useful tool for tracking my grocery expenses.

As a relatively frugal person who likes tracking each dollar I spend, I found it really helpful and thought maybe other people like me also want to analyze their spending behaviour when making groceries.
That’s why I am here, asking for your honest opinion.

Would an app like this be useful to you?

Do you believe it could help you analyze and manage your grocery spending effectively?

Let me know in the comment section below if you would actually use an app like this or maybe there is already one like that with a good design and all the features mentioned above that you’ve been using for a while.
Feel free to share any additional ideas or suggestions you may have for improving this app.

Your feedback is greatly appreciated.

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Feel free to follow me on GitHub and Medium!

Adios, mi amigos)

CSS Stacked Cards: Simple and Easy way

Kairat — Sat, 27 Jan 2024 23:45:09 +0000

Probably you've seen this cool effect on some websites where images stack on top of each other, you liked this animation and were wondering how it can be done.

Do I need to use the animation library for that? GSAP? Or maybe Framer Motion.

And the actual answer is no. It can be done with just plain CSS!

Here is the code sample:



<div>
    <div id="layer1" class="bg-red-400  h-screen w-screen flex justify-center items-center top-0 sticky">
        <p>CSS</p>
    </div>
    <div id="layer2" class="bg-orange-400 h-screen w-screen flex justify-center items-center top-0 sticky">
        <p>is</p>
    </div>
    <div id="layer3" class="bg-blue-400 h-screen w-screen flex justify-center items-center top-0 sticky">
        <p>not</p>
    </div>
    <div id="layer4" class="bg-green-400 h-screen w-screen flex justify-center items-center top-0 sticky">
        <p>awesome!</p>
    </div>
</div>

In this example I used TailwindCSS, but you can use whatever you want - plain CSS or any other library.
So, the most important properties here are these two:

top-0 that sets top to 0
sticky that sets the position of the card to sticky

And not to forget, all cards are contained within the div - the containing block-level ancestor.

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

Email Obfuscation: Best Methods

Kairat — Fri, 19 Jan 2024 22:43:25 +0000

If you are a front-end developer, there is a high chance that you've been asked to integrate contact details on the website at least once. It might be social media links, phone numbers, emails, etc.

And it sounds easy, right? What could go wrong? It should be simple - just use the "a" element and you should be good.

<a href="mailto:randomemail@address.com">Email</a>

Let's say you do that and a month later you end up having dozens of spam emails coming from random places that you never signed up for.
Well, do not be surprised cause the approach we used has one problem - our email address became an easy target for crawlers, spam bots and phishing attacks.

What should you do instead?

To avoid this problem, there is a working (not 100%) solution called email obfuscation.

What is that?
So, let's start with the concept of obfuscation.
If you've never heard of it, basically the idea behind it is to make the code difficult for computers / human beings to understand.
In case of email obfuscation, we try to confuse those bots & crawlers and make our email unreadable for them or difficult to understand.

But as I mentioned previously, it not only makes it difficult for computers but also for users as well and can result in a worse user experience.

That's why our solutions should be relatively effective against bots but at the same time accessible for the end-users and not worsen overall UX.

What are the techniques?

Before looking into that, I need to mention that sometimes we do not need email to be a link. Sometimes it can be just part of the plain text and not clickable at all.
Nevertheless, we have to protect these email addresses as well, but just using different techniques.

First, let's start with the clickable emails.

One way would be to use HTML (Character) Entities. An HTML entity is a piece of text ("string") that begins with an ampersand ( & ) and ends with a semicolon ( ; ). For example, this s entity represents "s".
Usually, they are used to display reserved characters (which would otherwise be interpreted as HTML code), or invisible characters (like non-breaking spaces).
In our case, we use entities to display the email address. Almost any web browser should be able to translate it back into readable format.

As an example, example@gmail.com would look like this: example@gmail.com

You can do this manually or using online tools (https://www.hcidata.info/obfuscate-email-address.htm)

While it might help against simple scrapers and bots, it is not the best solution as it is super easy to decode.

Another way to obfuscate our email would be to involve JavaScript in the process.
Let's say we have the following HTML element:

<a id="emailLinkID" href="">Send me an Email</a>

Then we can run the JS script to find that tag by ID and modify its href attribute by decoding base64 encoded string.



const emailForm = document.getElementById("emailLinkID");
emailForm?.setAttribute("href", "mailto:".concat(window.atob(window.btoa("example@gmail.com"))));

So, now when the browser finishes loading and the script above executes, the href attribute is populated with the actual email address.

You can use other encryption algorithms like ROT13 or Caesar cipher.

The only major downside of this approach is that it involves JavaScript and if the user disables JS on their browser, the link will be empty.

Now let's consider those cases where email does not need to be clickable.

The simplest but at the same time almost worthless solution would be to use HTML comments.

Email address:  email@gmail.com
For the end-user comments won't show up and regular email address will be seen.

A more effective solution would be to use CSS.
Email Address example@hiddentexthere.com 

Then using CSS hide the content of the "b" tag.



p.b {
  display: none;
}

Again the final user will not see "hiddentexthere" part.

Another solution would be to put an email address into an image which will make bots' lives more difficult but at the same time from a usability point of view, it is super inconvenient for the user as he/she won't be able to just copy it.

Another quite effective way of obfuscation is the simple replacement of “@” with “AT” and “.” with “DOT”. But again, not good for the final user.

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

Unlocking Seamless Deployments: Crafting a CD Pipeline with HyGraph Webhooks, Serverless Functions, and GitHub Actions!

Kairat — Tue, 14 Nov 2023 23:23:30 +0000

So, your company relies on HyGraph as the primary Content Management System (CMS), a tool you find enjoyable to work with due to its user-friendly interface, GraphQL API, and seamless integrations with Cloudinary, AltText, and more.

You successfully built and deployed your static website, everything works great, and everyone is happy.

A few days later, someone at your company messages you and says that they updated content on HyGraph and published it, but they don't see any changes on the website.

And it is the moment when you realize that you need a CD pipeline: all your pages are statically generated at the build time causing the new content from HyGraph not to show up on the website, and you have a choice: you can do it manually - rebuild the pages and upload to your hosting platform or automate it.

What should you choose?
As developers, we always go with the second option - even if it takes more time than doing stuff manually (not in this case, at least in the long term).

So, you decided to automate it.

What do you need for this?

First, you need a GitHub Actions workflow that will rebuild our website with the latest data from HyGraph.
If you are not aware of Github workflows, basically it is an automated process that will run one or more jobs when triggered.
You can read more about it in this article by GitHub.

A simple GitHub workflow can look something like this:

name: Name of the workflow

# when it should be triggered
on:
  - push
  - workflow_dispatch

jobs:
  # job name
  cypress-run:
    # running tests

  # another job
  build:
    # steps to build your pages

The logic for the build process would vary based on what library/framework you used.

For us, the most important part here is when the workflow is triggered. There are a lot of ways to trigger the GitHub Actions workflow - in our example, it starts on the push to the repository and on workflow dispatch.

And probably you have a question - what the heck is workflow dispatch?

Basically, workflow dispatch allows us to trigger workflow manually using the GitHub API, GitHub CLI, or GitHub browser interface.

So, how it can help us achieve our goal? As mentioned above, we can use Github API to trigger our workflow and that is exactly what we gonna do.

We have a GitHub Actions workflow that rebuilds our pages and can be manually triggered using Github API. Now what?

The next step would be to somehow trigger it with the help of GitHub API.
How? By making a POST request to this endpoint:
https://api.github.com/repos/OWNER/REPO/actions/workflows/WORKFLOW_ID/dispatches

You have to replace OWNER, REPO and WORKFLOW_ID with the actual data (workflow id should be the name of a file (with extension) - deploy.yaml as an example)

Plus, you have to provide one required body parameter - ref which is the git reference for the workflow. The reference can be a branch or tag name.

It is the simplest way we can use this endpoint but if you want to know more - feel free to go through GitHub docs on this particular topic.

Now, we know the endpoint we should use to trigger the workflow and the only problem left is where to make this call.

Probably while working with HyGraph you've seen the "Webhooks" tab in the menu but never got a chance to learn more about it.

Probably it would be the perfect time to do this!
Why?

HyGraph WebHooks is a method designed to detect changes that happen to content within your project.
This means that we can be notified whenever new content is published, or existing is updated, by subscribing to these events, and then perform our own custom logic.

Cool, does it basically mean that we can just use Webhook, send a POST request to the Github API endpoint we discussed above and that's it, everything would work like a charm?

In an ideal world, yes! But there is one minor problem that won't make it as easy as you think.

As was mentioned above, when sending a POST request to this endpoint (https://api.github.com/repos/OWNER/REPO/actions/workflows/WORKFLOW_ID/dispatches), we also need to send a body with the ref parameter set to either tag or branch name.

Maybe you worked with other CMS tools such as Contently, and DatoCMS, and with these tools, this logic would work perfectly fine cause their webhooks implementation has a feature of customizable payload - which means you can modify the body (data) you send with your requests.

Unfortunately, in HyGraph there is no such feature yet (but I got in touch with one of their sales representatives and he confirmed that it should be released soon, but did not give a specific timeline).

Because of this, we need a Serverless Function that will serve as a middleman whose entire purpose would be to add this "ref" parameter to the body of the request and actually make this POST request

I hope you are all familiar with the concept of serverless functions, but if not, you can read this short article by Cloudflare.

You can use AWS Lambda Functions, Azure Functions, GCP Cloud Functions or any other implementation of serverless function.
Up to you!

I am going to use Azure Functions, and here is an example of how the implementation of that function should look like:

 const axios = require("axios");
  let data = JSON.stringify({
    ref: "main",
  });

  let config = {
    method: "post",
    maxBodyLength: Infinity,
    url: "https://api.github.com/repos/OWNER/REPO/actions/workflows/WORKFLOW_ID/dispatches",
    headers: {
      Accept: "application/vnd.github+json",
      "Content-Type": "application/json",
      Authorization:
        "Bearer GITHUB_PERSONAL_ACCESS_TOKEN",
    },
    data: data,
  };

  axios
    .request(config)
    .then((response) => {
      context.log(JSON.stringify(response.data));
    })
    .catch((error) => {
      context.log(error);
    });

  return { body: `OK` };

My Azure Function uses Node.js as a runtime and thus I use Axios to make a request. But you can use any other programming language based on your preferences.

The most important part here is that we construct the body of our request and include the "ref" parameter set to "main" (branch name, but can tag name as well).

Don't forget to change the OWNER, REPO, WORKFLOW_ID and GITHUB_PERSONAL_ACCESS_TOKEN with the actual data.

Now, when you have your serverless function ready and live, it's time to configure HyGraph Webhook.

Go to the "Webhooks" tab and click the "Add" button.
Configure your webhook - give it a name, and a short description of what it does, choose the method, and provide the serverless function public URL (make sure, your serverless function is accessible)
Choose which models, actions, and stages should trigger this webhook
Click the "Add" button at the top.

And here we go! You just created your HyGraph Webhook.

Now, your CD pipeline is ready!

Whenever you make a change to your content in the HyGraph and publish it (or unpublish), it triggers a webhook that makes a call to the serverless function, which then sends a POST request to the Github API and triggers workflow which fetches the latest data from HyGraph, rebuilds the pages and pushes updated content to the hosting platform.

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

Setting up TWS & IBC on EC2 instance

Kairat — Tue, 11 Oct 2022 07:15:09 +0000

Probably if you are a developer who was always interested in the stock market or any other type of investing, you have at least once thought about creating some sort of system that will help you get the market data and then use it for analysis

Especially these days when the stock market is crashing, the Japanese yen and British pound are hitting record lows against the dollar, etc.

Probably this is you when you open your portfolio

And because you definitely do not want to be in such a situation, you made a research on what kind of options you have to get the market data and found something called IB Trader Workstation (or TWS)

You start digging more and turns out that this TWS is actually exactly what you need - it gives you access not only to historical data but real-time too. Moreover, you get access to analysis tools, interactive, customizable charts, you can execute orders, etc.

Sounds quite interesting, right?
So, you go and install it on your local machine, play with it for a week as you absolutely loved it

Then you finally want to start coding and you stumble into TWS API which is a simple interface to TWS through which you can do a lot of stuff - from automating your trading strategies & requesting market data to monitoring your account balance and portfolio in real-time.

And the main requirement for this API to work is a running instance of Trader Workstation app (or IB Gateway, but we are not here to talk about it)

As TWS been already installed on your machine, you start writing code using TWS API - either its native version or any third-party libraries such as ib_insync.

You test your code and it works perfectly fine!

But then you start thinking that running TWS on your local machine is not a long-term solution and you decide to switch everything to the cloud

You think about which cloud platform you should use and decide to go with AWS, cause Jeff Bezos is the best

The first thing you have to do is launch the EC2 instance which we will use to run our Trader Workstation app

We are going to use Ubuntu as our AMI and for the instance type we will choose t2.medium (cause if you choose a less powerful instance type, TWS will work a little bit slow)

I suppose now you have a running EC2 Ubuntu instance with the instance type mentioned above and now let's start working on the installation of TWS on this EC2 machine

First, let's connect to your EC2 instance.
One of the easiest ways is through using SSH (you also need to have access to the file with the key pair that you used when creating the EC2 instance)

Choose your EC2 instance and click on the "Connect" button

From here, choose "SSH client" and just copy the example code

Then, use the terminal of your choice, and run the copied command (make sure that you run this command from the same folder where your key file is located)

You successfully connected to your EC2 and now let's run the following general commands



sudo apt update
sudo apt -y upgrade

After that, we have to install 2 additional things: XVFB and VNC server

XVFB - is an X server that can run on machines with no display hardware and no physical input devices. It emulates a dumb framebuffer using virtual memory.
In simple words, we will use it to add GUI to our EC2 instance.

VNC - is basically a screen-sharing system that is used to remotely control another computer.
It works on a client/server model which means that we install the VNC server on the remote computer - the one we want to control (in our case the EC2 instance) and on our local machine we install the VNC Client that we use to control another computer

To install XVFB run this command:



sudo apt install -y xvfb

As our VNC server, we will use x11vnc which allows us to view remotely and interact with real X displays (i.e. a display corresponding to a physical monitor, keyboard, and mouse) with any VNC viewer.



sudo apt install -y x11vnc

Then, run this command to start virtual frame buffer



/usr/bin/Xvfb :0 -ac -screen 0 1024x768x24 &

When you run this command, you should get the ID of the Xvfb background process

To start the VNC server, we will run this command where we specify 2 passwords - full access (keyboard + mouse) and second is view only.



/usr/bin/x11vnc -ncache 10 -ncache_cr -viewpasswd view_only_password -passwd full_access_password -display :0 -forever -shared -bg -noipv6

Also, we define what server display it should connect to (in our case, it is :0).
noipv6 option states that it should not try to use IPv6 for any listening or connecting sockets.
The shared option allows more than one viewer to connect at the same time and the forever option tells our VNC server to keep listening for more connections rather than exiting as soon as the first client disconnects

Now VNC server is listening on port 5900 and we can connect to it using any VNC client you want

I am going to use TightVNC but there are many other options such as TigerVNC, UltraVNC, etc.

Before connecting to our server, we have to make a few adjustments to the security group assigned to our EC2 instance.
Basically, we need to allow inbound traffic to port 5900 and we can easily do this by going to the AWS console, choosing our instance's security group and editing its inbound rules

Here we add a new rule of type Custom TCP and define our IP as a source

After that, we connect to the VNC Server using TightVNC Viewer.
In the hostname field, we enter the public DNS of our EC2 instance

And we use the full access password that you entered when starting the VNC Server

After connecting to our VNC server, let's make our life a little bit easier and install metacity - a lightweight window manager.



sudo apt install -y metacity

And run this command:



DISPLAY=:0 metacity &

Now we are ready to install the Trader Workstation app.
First, let's install offline TWS for Linux on our local machine

Offline TWS for Linux

After installation is complete, we will copy this file to our EC2 instance.

In order to do that, we will use the scp CLI tool that allows you to securely copy files and directories between two locations



scp -i /path_to_key_file tws-latest-standalone-linux-x64.sh ubuntu@public-ip-address-of-ec2-instance:

Do not forget to add ":" at the end of a public IP address

This command should copy the installation file from our local machine to the EC2 instance (to the home directory)

We can check this by running the ls command:

Now let's make this file executable



chmod a+x ~/tws-latest-standalone-linux-x64.sh

And as we have Xvfb installed, we can run the TWS installer on display ":0"



DISPLAY=:0 ~/tws-latest-standalone-linux-x64.sh

In TightVNC Viewer, you can see that the installation window is opened

Follow the steps to complete the TWS installation.
When you finish installation, you can run the ls command

As you can see, the Jts folder was created which contains all TWS-required files

Now you just have to use TWS credentials to log in and configure TWS as you want

And now you have the Trader Workstation app hosted on an EC2 instance!

So, let's say you've been using it for a few days and noticed that sometimes your TWS loses connection and asks for manual authentication
And it is definitely not the way you wanted it to work

Wouldn't it be perfect if there were a tool that can run TWS in 'hands-free' mode (without any manual intervention), right?

I agree that it would be nice to have such a tool and fortunately for us, there is one called IBC

You can follow the link above and read more about what IBC can do, but for our purpose, we will use it to start the TWS app and login automatically in case our TWS loses connection

First, we need to install it by running this command:



wget https://github.com/IbcAlpha/IBC/releases/download/3.14.0/IBCLinux-3.14.0.zip

Then, let's install unzip command (if you don't have it installed already)



sudo apt install -y unzip

Before extracting the contents of the downloaded ZIP file, I want you to create a new folder - /opt/ibc/

We do this because several IBC script files assume the default paths used for particular files (config file, TWS & IBC program files) and by using these default paths we will minimize customizing the configuration file and the shell scripts.



mkdir -p opt/ibc/
cd opt/ibc

Now let's unzip a file



unzip ../../IBCLinux-3.14.0.zip

Run these 2 commands to make sure all .sh files are executable:



chmod u+x *.sh
chmod u+x scripts/*.sh

Let's go back to the home directory and create another new folder - ibc where we will store the configuration file



cd ../..
mkdir ibc

Copy configuration file from /opt/ibc to the newly created folder



cp opt/ibc/config.ini ~/ibc/

Now we have to make a few changes to that config file (use the editor of your choice):



vi ibc/config.ini

Here we have to change the username & password fields

Save the changes and let's go back to the /opt/ibc/ folder
Here open the twsstart.sh file using an editor of your choice



cd opt/ibc/
vi twsstart.sh

For the trading mode, specify the one you need - either live or paper (I will use live)

And also, we need to change the TWS version.
We can find the TWS major version number by running the TWS app, clicking the Help button on the top navigation menu, then About Trader Workstation option

In the displayed information you'll see a TWS version

Going back to twsstart.sh file, modify the TWS_MAJOR_VRSN property
And for the IBC_PATH parameter add ~ before /opt/ibc

Save the changes and let's try to start IBC!
Before doing this, make sure that the previously started TWS is no longer running



DISPLAY=:0 ~/opt/ibc/twsstart.sh

After running this command, you should see TWS been started in VNC Viewer

And if you want to make IBC run every hour to check whether TWS lost connection and requires re-login, we can use crontab and schedule a cronjob that will run this command (DISPLAY=:0 ~/opt/ibc/twsstart.sh) every hour

Run this command:



crontab -e

And add the following line to the end of the file (you can adjust the time according to your needs)



0 * * * * DISPLAY=:0 /home/ubuntu/opt/ibc/twsstart.sh

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, I want to mention that I used this amazing article as a reference and my post is just a shorter and easier version of that article

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

Migrating data from GCS to AWS S3 using AWS DataSync

Kairat — Mon, 29 Aug 2022 06:59:00 +0000

Let's say your company has been using the Google Cloud Platform for a very long time.

And suddenly, one day, your company's Solutions Architect decided to switch to AWS cause Amazon offers more services (maybe it sounds like an unrealistic situation but let's just imagine)

Because of that, you've been given a task to migrate all the data from GCS (Google Cloud Storage) to AWS S3

You started exploring different possible solutions and found something called AWS DataSync

It is actually a good question.

So, DataSync is one of many AWS Services and it allows you to simplify, automate and secure the data transfer process between different storage systems

Sounds good, right?
But why would you use it and pay for it, when you can build your own solution for that purpose.

Let's start with that building such solution will require a lot of time, and even if you will accomplish that, there is no guarantee that your solution will be cheap, secure and easy to scale.

Therefore, maybe it will be just easier and better not to reinvent the wheel and use DataSync that already has such features as data encryption, validation, monitoring, task scheduling, auditing, and scaling. Also, I think it is important to mention that DataSync is available at 24 AWS Regions!

Hopefully, I convinced you to use DataSync.
The next question you probably want to ask me is how it works.

And before I will answer this question, I want to introduce you to a few very important concepts that I will refer to in the rest of this article.

One of them is DataSync Agent which is a VM (virtual machine) owned by you and used to access your self-managed storage and read data from or write data to it (self-managed storage can refer to on-premises or in-cloud storage systems. Transfers with these kinds of systems require using a DataSync agent)

It can be deployed on VMware, Hyper-V, KVM or as an AWS EC2 instance.

AWS DataSync provides several types of agents for different storage environments.
For example, you can use a VMware agent to transfer data from an on-premises file system.
And if you're copying data from a cloud-based file share that isn't an AWS storage service, you must deploy your DataSync agent as an Amazon EC2 instance (which is our case)
If you are copying data between AWS storage services, you do not need DataSync agent at all.

The other important concept I need to mention is a location which basically identifies where you're copying data from or to.
Each DataSync transfer has a source and destination location.

And last but not least is the task which is simply a DataSync transfer that identifies a source and destination location along with details about how to copy data between those locations.
Task execution is just one individual run of a DataSync task.

And now, when we are all familiar with the terminology, I am ready to answer the question - "how does DataSync work", and I'll do that by going through the process of migrating data from GCS to AWS S3.

First, we need to launch an EC2 instance which will be used to deploy the DataSync agent (we do this as we use an in-cloud storage system - GCS, otherwise we would use an on-premises hypervisor such as Hyper-V or VMware)

As we know, the EC2 instance requires AMI to be chosen.
(for those, who do not know, AMI stands for an Amazon Machine Image and it is a template that contains the software configuration (operating system, application server, and applications) required to launch your instance. But we are not here to talk about AMI, therefore here is the link if you want to know more about it)

The reason I mentioned AMIs is that we will launch our EC2 instance based on the image provided by DataSync. This image contains the DataSync VM image.

Agent AMIs vary by region, thus you have to choose the right one for your region.

In order to get the data about the latest DataSync AMI for a specified region, run this command (AWS CLI has to be installed on your machine)



aws ssm get-parameter --name /aws/service/datasync/ami --region yourRegion

The result of running this command will be an object containing information about the DataSync image - AMI ID (value), version, ARM, etc.

Now, use the following link and replace 2 parameters - source-region and ami-id with your source file system region and AMI ID that we got by running the previous command



https://console.aws.amazon.com/ec2/v2/home?region=source-region#LaunchInstanceWizard:ami=ami-id

By going through this link, "Launch an instance" page will be opened

Enter the name of your EC2 instance.
Do not modify the "Application and OS image" section as AMI has already been chosen
For the "Instance type", you have to remember that the instance size must be at least 2xlarge (recommended ones are m5.2xlarge (for tasks to transfer up to 20 million files) or m5.4xlarge (for tasks to transfer more than 20 million files)
I will choose t2.2xlarge (32 GiB, 8vCPUs, 64-bit)
Create a new key pair and save it
In the "Network settings" section, you can edit your VPC and subnet.
Enable Auto-assign Public IP
For the "Security group" section, make sure that the selected security group allows inbound access to HTTP port 80 from the web browser that you plan to use to activate the agent
In "Configure storage" section, ensure that there is at least 80 GB of disk space
Click the "Launch Instance" button

The initial state of the EC2 instance will be pending.
You have to wait until its status changes to running.
When your instance is running, it's assigned a public DNS and IP address that we will need later

Now, as we deployed our DataSync agent, we need to activate it cause activation securely associates the agent that you deployed with your AWS account

In order to do that, go to the AWS DataSync console, choose Agents on the left-side bar and then click the "Create agent" button.

On the "Create agent" page the first section you see is "Deploy agent"

As we deployed our DataSync agent as an EC2 instance, choose "Amazon EC2"
The second section is the "Service endpoint" that the DataSync agent uses to communicate with AWS.
You have 3 options - public, VPC or FIPS.
Also, I need to mention that after you choose a service endpoint and activate the agent, you won't be able to change the endpoint

For simplicity, I am going to choose the public service endpoint.
It means that all communication from your DataSync agent to AWS occurs over the public internet

The last section of this page is called "Activation key"

To obtain the agent's activation key, you have 2 options - do it automatically or manually.

We will do this automatically, and thus we need to specify the agent's address (to get the IP address, go to your EC2 running instances, choose the one we deployed earlier in this tutorial, and copy its public IP address)

Put copied IP to the "agent address" field.

After that, you can press the "Get key" button.
Your browser will connect to the IP address and get a unique activation key from your agent

If activation succeeds, you will see a "Successfully retrieved activation key from agent" message with the activation key itself.

Then just type the agent name (not required) and press the "Create agent" button

The DataSync agent was successfully activated!

We are almost done, guys!
Just a few small steps left - creating source & destination locations and creating the task itself.

Let's start with the configuration of a source location.

On the DataSync console, go to "Locations", and click the "Create location" button

For the location type, choose "Object storage"

A lot of new fields appeared.

Choose our previously activated agent in the "Agents" field
For "Server", type “storage.googleapis.com”
For "Bucket name", type the name of the GCS source bucket

If you want, you can specify the folder that will be used for the data transfer (I will leave it empty)

In the "Authentication" section, you have to specify HMAC credentials that include access and secret keys.

DataSync agent uses an HMAC credential to authenticate to GCP and manage objects in the Cloud Storage bucket

We are not going to talk about HMAC keys but here is the link to how you can create one

Enter your access & secret keys, and you are ready to press the "Create location" button

Now, let's go through the same process and create a destination location which is our AWS S3 Bucket.

Click the "Create location" button on the "Locations" page.

For the "Location type" field, choose "Amazon S3"
For the "S3 bucket", select the destination AWS S3 bucket
For the "Storage class" you can leave the default value - Standard.
In the "IAM role", choose a role with appropriate permissions

(DataSync needs to access the AWS S3 bucket in order to transfer the data to the destination bucket. This requires DataSync to assume an IAM role with appropriate permission and a trust relationship - here is the link)

And finally, we just need to create the AWS DataSync task

On the DataSync console, go to "Tasks", and click the "Create task" button

In step 1 - "Configure source location", select "Choose an existing location"

For "Existing locations", choose our previously created GCS location

In the second step - "Configure destination location" do the same as in the previous step, but for "Existing locations" choose the S3 location

In the third step - "Configure settings", enter the task name

In the "Task execution configuration" section, for "Verify data" choose the needed option (I will leave the default one)

In the "Data transfer configuration" section, leave everything as default values (but you can adjust it if needed)

In the "Schedule" section, choose a schedule you want this task to be executed on

Go to step 4 - "Review", and click the "Create task" button (wait for Task status to be Available)

And that's it, guys.

It was really long article but I hope you made it to the end and were able to successfully run your first AWS DataSync task!

I would appreciate it if you could like this post or leave a comment below!

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

React Fragments in 5 minutes (with examples)

Kairat — Sun, 08 May 2022 04:16:02 +0000

And you think to yourself: "Oh God, what have I done wrong again?"

But luckily for you (and for all of us - React developers), this time the problem is super simple.

Let's take a look at the code that caused the problem.

import React from "react";

const Fragment = () => {
  return 
  <div>Fragments</div>
  <p>in React</p>
  ;
};

export default Fragment;

As you can see, we are trying to return two JSX elements from our Fragment component.

And, as you probably understand, this is the root of our problem.

So, why?

Isn't it a common situation when you need to return multiple elements from a component?

And you are right - this is a common pattern in React.

But you have to remember:

Component must return only one parent element and not more, not less.

What can we do about this?

So, one solution is to wrap it in a "div" element.

Let's take a look at whether it works or not!

import React from "react";

const Fragment = () => {
  return (
    <div className="wrapper">
      <p>Fragments</p>
      <p>in React</p>
    </div>
  );
};

export default Fragment;

Hooray! It worked!

But is this the best solution?

Nah!

There are several problems that come with it.

First of all, we add an extra node to our DOM. It takes extra space in memory.

Secondly, it's just irrelevant. We need it only for our JSX to work. Kind of ridiculous, right?

And in addition to that, sometimes this div wrapper can even break our layout and lead to invalid HTML to be rendered!

You probably think, how this innocent div can break your HTML?

Let me show you a quick example that will demonstrate that!

Let's imagine that we wanna display a table.
So we define "table", "tbody", "tr" elements.

import Fragment from "./fragment";

function App() {
  return (
    <div className="App">
      <table>
        <tbody>
          <tr>
            <Fragment />
          </tr>
        </tbody>
      </table>
    </div>
  );
}

export default App

And let's say, we want a separate component for our data cells ("td" elements).

import React from "react";

const Fragment = () => {
  return (
    <div className="wrapper">
      <td>Team </td>
      <td>Points</td>
      <td> Wins</td>
    </div>
  );
};

export default Fragment;

We already know that we cannot return multiple values from a component.

Thus we will wrap our "td" elements in a div with class "wrapper" (no styles applied to this class, just for demonstration purposes)

Now let's check if it works or not!

Seems like it works but damn...
Look at the console ...

We cannot place "td" inside of a div.
And "tr" cannot contain the div element.

Q.E.D.

So, what should we do instead?

You guessed it right - we can use Fragments!

A fragment is just a syntax that allows you to group a list of children (as our div did) but...

Super important, Fragment does not add an extra node to the DOM.

Okay, that sounds like a win-win, right?

How to use it?

Pretty easy !

import React from "react";

const Fragment = () => {
  return (
    <React.Fragment>
      <p>Fragments</p>
      <p>in React</p>
    </React.Fragment>
  );
};

export default Fragment;

All we have to do is use React.Fragment instead of div.

Also, I need to mention that there is another form of React Fragment - short syntax that looks like an empty tag.

import React from "react";

const Fragment = () => {
  return (
    <>
      <p>Fragments</p>
      <p>in React</p>
    </>
  );
};

export default Fragment;

Both of these code examples will result in such DOM:

As you can see no additional JSX elements have been added to the DOM!

Is there any difference between React.Fragment and the short syntax?

Actually, there is one small difference

Look at this code snippet:

import React from "react";

const Fragment = () => {
  const arrayOfNumbers = [1, 2, 3, 4, 5];
  return arrayOfNumbers.map((number, index) => (
    <>
      <p>{number}</p>
      <p>{index}</p>
    </>
  ));
};

export default Fragment;

We iterate through the array and for each element return 2 paragraphs (value itself and its index).

We put these paragraphs in an empty tag (short syntax of React.Fragment)

Everything works fine but ...
We got this error in the console ...

If you read my previous post about React keys, you know what it's all about and how to deal with it.
If not, check it out!

So, as you understand, we have to add the key to our parent element.

And here is the problem.

We cannot do that with the short syntax of React.Fragment.

Instead, we have to use React.Fragment and give it a key attribute.

import React from "react";

const Fragment = () => {
  const arrayOfNumbers = [1, 2, 3, 4, 5];
  return arrayOfNumbers.map((number, index) => (
    <React.Fragment key={index}>
      <p>{number}</p>
      <p>{index}</p>
    </React.Fragment>
  ));
};

export default Fragment;

Brief conclusion:

Fragments can be used whenever you need to return multiple JSX elements from the component.
You should be using them instead of wrapping everything in a div because React.Fragment does not add an extra node to the DOM. There are 2 forms of it: full and short syntax.
The only difference between them is that you cannot use key attribute (if you need one) with short syntax.

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

Everything you need to know about keys in React (with examples)

Kairat — Sat, 19 Mar 2022 22:50:36 +0000

I can bet my life that you've seen this warning at least once while building React applications.

Those who say "no" ...

So, back to the problem...

Maybe you thought:
What on earth is a "key"?
Why it should be unique?
And why should I even care, it's just a warning?

So many questions and I hope I will be able to answer all of them.

Let's start with the basics ...

What is a key?

According to the React documentation,

key - is just a special string attribute that must be included when creating arrays of elements.

What did we get from this definition?

Weell, first, we now know that key is of type string and
secondly, we should use it when we work with arrays!

How to use them?

Super simple!
Let's take a look at this code snippet:



const [soccerTeams, setSoccerTeams] = useState([
  "Chelsea",
  "Real Madrid",
  "Liverpool",
  "Juventus",
]);

return (
  <div className="App">
    {soccerTeams.map((team) => (
      <p> {team}</p>
    ))}
  </div>
);

As you can see we just loop through the array and display the content of each element.

And this code will produce the warning you've seen in the beginning (that says "each child should have a unique key and blah blah blah")

So, the obvious question is how to get rid of this red little dude with the help of keys, right?

Actually, it's not that hard at all
All you have to do is assign the "key" attribute to array elements that are inside the map() and give it a value of type string.

Probably, your next question could be: what should we provide as a "value" to it?

Hmm, let's think about it ...

As we all know, each element in the array has its own index and this index is always unique.
So why not use it?

Okay, let's try ...



const [soccerTeams, setSoccerTeams] = useState([
  "Chelsea",
  "Real Madrid",
  "Liverpool",
  "Juventus",
]);

return (
  <div className="App">
    {soccerTeams.map((team, index) => (
      <p key={index}> {team}</p>
    ))}
  </div>
);

We just added the "key" attribute with the value of index to each array element.

As you can see, it worked!
There is no more this freaking warning!

But perhaps you still ask yourself:

Why did we get this warning in the first place and why does React care so much about it?

Well, have you ever heard anything about reconciliation in React?
I guess, it sounds like something familiar, right?

This post is not about reconciliation and thus we won't spend much time on it but briefly, it is a process that helps React to decide whether Virtual DOM should be updated or not (when component's state changes).

How do keys relate to all this reconciliation stuff?

Basically, when the component re-renders, React compares new keys against the old set of keys and identifies which items have been modified, added, or deleted.
And based on that, updates Virtual DOM.

And that is actually all you have to know about keys in React!
Yeah, dead simple ...
Don't forget to follow me on GitHub and Medium ...

Ookay, okay guys)
Of course, it was just a joke!

Remember, it is React and it just can't be that simple and easy.
There is always a catch!

Let's take a look at this code.



const [soccerTeams, setSoccerTeams] = useState([
  "Chelsea",
  "Real Madrid",
  "Liverpool",
  "Juventus",
]);

const deleteItem = (index) => {
   const tempArray = [...soccerTeams];
   tempArray.splice(index, 1);
   setSoccerTeams(tempArray);
 };

return (
  <div className="App">
    {soccerTeams.map((team, index) => (
      <div key={index}>
        <p> {team}</p>
        <button onClick={() => deleteItem(index)}>
          Delete from array
        </button>
      </div>   
    ))}
  </div>
);

Everything is almost the same as the previous one - we use "index" as "key", no more warning.

But we added a button so we can delete the particular items from the array.

So, what gonna happen when we delete the item from an array?

Logically, it should just erase this particular node from DOM and that's it, right?

Other items have already been rendered, and why update them

Good logic, but let's see what is really going to happen.

Can you see those purple flashes in dev tools?

They indicate that something has been updated inside the DOM.

And as you can see with your own eyes, those elements that have already been in DOM were updated too.

Why?
Let's go through this process very carefully.

We have an array that consists of 4 elements.
The first element has index 0, the second one - index 1, and so on.

Then we delete the first element.
What happens?

Our second element that had index 1, now has index 0.

And as you already understand, the key is also updated because we assigned it to be equal to the index (that was updated) of our element.

The same nonsense occurs when we add a new element to the array.

Also, I want to bring to your attention the fact, that when we add a new element to the end of the array or delete an item from the end, nothing is gonna happen and everything will work totally fine (DOM won't update other elements)

But that's not always the case.

Therefore, keys always should be stable.

How to make keys constant?

Very good question.

One of the best ways is the usage of ID as a key's value.

Most of the time when we fetch data from some sort of database, items we fetch have their own unique IDs.
And these IDs can be used as keys.

Let me show a quick example of how it can be done



const [soccerTeams, setSoccerTeams] = useState([
    { team: "Chelsea", id: "667" },
    { team: "Liverpool", id: "545" },
    { team: "Juventus", id: "1393" },
    { team: "Real Madrid", id: "432" },
  ]);

  const deleteItem = (index) => {
    const tempArray = [...soccerTeams];
    tempArray.splice(index, 1);
    setSoccerTeams(tempArray);
  };

  return (
    <div className="App">
      {soccerTeams.map((element, index) => (
        <div key={element.id}>
          <p> {element.team} </p>
          <button onClick={() => deleteItem(index)}>
            Delete from array
          </button>
        </div>
      ))}
    </div>
  );

Now, elements in our fictional database have their own IDs.

When we loop through the array we use this "id" property as a value for the "key" attribute.

But the main question is what's gonna happen when we will delete the element from an array.

Intrigued?

Yo ho ho ...
No more purple flashes !
Only parent div updated as one element was removed in it, but other list items were not re-rendered because we use constant ID as a key.

So even we delete an item, the keys for other elements stay the same as we don't use indexes as their values.

Amazing! Problem solved!

Buut ... what if your data does not have IDs?

Good question, because not all data has IDs.

But what if can generate one?

For example, let's try to use a popular ID generation tool called UUID.



const [soccerTeams, setSoccerTeams] = useState([
    { team: "Chelsea" },
    { team: "Liverpool" },
    { team: "Juventus" },
    { team: "Real Madrid" },
  ]);

  const deleteItem = (index) => {
    const tempArray = [...soccerTeams];
    tempArray.splice(index, 1);
    setSoccerTeams(tempArray);
  };

  return (
    <div className="App">
      {soccerTeams.map((element, index) => (
        <div key={uuidv4()}>
          <p> {element.team} </p>
          <button onClick={() => deleteItem(index)}>
            Delete from array
          </button>
        </div>
      ))}
    </div>
  );

In this example, we generate value for the key using the UUID() function.

Oh no ...
We are back to our initial problem.
DOM is updated every time we delete the item from an array.

I guess you already understand why.

Every time a component re-renders, a new ID is generated and assigned to the key.

So, React thinks it is a brand new element, while it's not.

But there is another way we can use UUID.



const [soccerTeams, setSoccerTeams] = useState([
    { team: "Chelsea", id: uuidv4() },
    { team: "Liverpool", id: uuidv4() },
    { team: "Juventus", id: uuidv4() },
    { team: "Real Madrid", id: uuidv4() },
  ]);

  const deleteItem = (index) => {
    const tempArray = [...soccerTeams];
    tempArray.splice(index, 1);
    setSoccerTeams(tempArray);
  };

  return (
    <div className="App">
      {soccerTeams.map((element, index) => (
        <div key={element.id}>
          <p> {element.team} </p>
          <p> {element.id} </p>
          <button onClick={() => deleteItem(index)}>
            Delete from array
          </button>
        </div>
      ))}
    </div>
  );

Here we use the UUID() function to generate ID for the id property.

This way everything works fine!

Is there another way?

Actually, yeah.
We can use some hashing tool to generate a hash from the object and use it as a key's value.



  const [soccerTeams, setSoccerTeams] = useState([
    { team: "Chelsea" },
    { team: "Liverpool" },
    { team: "Juventus" },
    { team: "Real Madrid" },
  ]);

  const deleteItem = (index) => {
    const tempArray = [...soccerTeams];
    tempArray.splice(index, 1);
    setSoccerTeams(tempArray);
  };

  return (
    <div className="App">
      {soccerTeams.map((element, index) => (
        <div key={hash(element)}>
          <p> {element.team} </p>
          <p> {hash(element)} </p>
          <button onClick={() => deleteItem(index)}>
            Delete from array
          </button>
        </div>
      ))}
    </div>
  );

Here we use the object-hash package, to generate a hash from the object and use it as a key.

As you can see, no problems over here!

But maybe it's not the best approach as hashes don't ensure uniqueness.

Moreover, if you have objects with the same contents, it can lead to a problem!

In the end, let me mention a few things we haven't touched before

Take a look at this code:



 const [soccerTeams, setSoccerTeams] = useState([
    { team: "Chelsea", id: "667" },
    { team: "Liverpool", id: "666" },
  ]);

  const [soccerTeams1, setSoccerTeams1] = useState([
    { team: "Juventus", id: "667" },
    { team: "Arsenal", id: "666" },
  ]);

  return (
    <div className="App">
      {soccerTeams.map((element) => (
        <div key={element.id}>
          <p> {element.team} </p>
        </div>
      ))}
      {soccerTeams1.map((element) => (
        <div key={element.id}>
          <p> {element.team} </p>
        </div>
      ))}
    </div>
  );

We have two different arrays.

And as you probably noticed their elements have the same IDs.

Will this cause a problem (like - two children have the same ID and blah blah)?

Nah! Keys could be the same for 2 different arrays

Just remember, keys must be only unique among siblings.

One more example and we are done, guys!

What's wrong with this code?



<div className="App">
   {soccerTeams.map((element) => (
     <div>
       <p key={element.id}>{element.team}</p>
     </div>
    ))}
</div>

We have a key, but in our console, we see the warning.
React is not happy!

The problem is simple - the key needs to go on the outermost returned element!

In our case - on the div element.



 <div className="App">
   {soccerTeams.map((element) => (
      <div key={element.id}>
        <p>{element.team}</p>
      </div>
   ))}
</div>

Hah! Problem solved!

Is there any default value?

Almost forgot) Yeah, there is.

According to React docs, if you choose not to assign an explicit key to list items then React will default to using indexes as keys.

But as we already went through this, you understand that maybe it's not a good idea!

Brief conclusion:

Keys are crucially important and you should not neglect using them.
In the best scenario, you should use IDs as values for keys, but if you do not have ones, you can use hashing, ID generation tools.
In some cases, it's totally appropriate to use indexes (if it's just simple data that won't change in the future)

And that's it, guys.

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!

Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)

Everything you need to know about React.memo (with examples)

Kairat — Sat, 05 Mar 2022 05:14:41 +0000

Probably you have already reached some level at React - you have an understanding of what state and props are, how to use basic React hooks - useState, useEffect.

And maybe, you've started to notice that sometimes your React components work very slow (especially the heavy ones with a lot of UI elements and other components)

And you started to ponder on how to fix it and optimize the performance...

After some research, you stumbled upon something called React memo.

You could ask yourself: What the heck is that ?

So, React memo is a HOC - higher-order component that allows you to improve the performance of your React app.
It skips rendering the component if passed props have not changed.

How does it work?

Super simple. A memo will just memoize the rendered output of the component and before the next render, it will compare props.
If nothing changed, the memo will just reuse the last rendered output.

Let me show you an easy example that will demonstrate the difference between the component wrapped into React.memo HOC and just plain component.

We have an "App" component that has 1 state variable "counter".

Also, it has 2 child components - PlainComponent (that is just a plain component that does not use React.memo HOC) and MemoComponent (that is wrapped into React.memo HOC)

function App() {
  const [counter, setCounter] = useState(1);
  return (
    <div className="App">
      <p> {counter}</p>
      <button onClick={() => setCounter(counter + 1)}> Set Counter</button>
      <div className="childComponents">
        <MemoComponent />
        <PlainComponent />
      </div>
    </div>
  );
}

const PlainComponent = () => {
  console.info("Child Component - no memo HOC");
  return (
    <div>
      <h3> Plain Component </h3>
    </div>
  );
}

const MemoComponent = React.memo(() => {
  console.info("Child Component - uses memo HOC");
  return (
    <div>
      <h3> Memo Component </h3>
    </div>
  );
}

So, when we will change the state of the "App" component (by incrementing a counter), it should cause its child components to re-render.

But as you can see, only the plain component re-rendered.

React.memo & props

But we haven't passed any props to our child components.

What if we gonna pass the prop to the Memo component and this prop will change.

Is it going to ignore this change or it will re-render the component and reflect the modification?

Let's take a look at another example!

We are going to use the same "App" and "MemoComponent" but this time, I added one more state variable to the App component - "passedProp".

This variable will change every time the remainder of our "counter" will be equal to 0.

And we gonna pass this prop to the "MemoComponent"

function App() {
  const [counter, setCounter] = useState(1);
  const [passedProp, setPassedProp] = useState(0);

  useEffect(() => {
    if (counter % 5 === 0) setPassedProp(passedProp + 1);
    // eslint-disable-next-line react-hooks/exhaustive-deps
  }, [counter]);

  return (
    <div className="App">
      <p> {counter}</p>
      <button onClick={() => setCounter(counter + 1)}> Set Counter</button>
      <div className="childComponents">
        <MemoComponent prop={passedProp}/>
      </div>
    </div>
  );
}

In MemoComponent we will just display passed prop

const MemoComponent = React.memo(({prop}) => {
  console.info("Child Component - uses memo HOC");
  return (
    <div>
      <h3> Memo Component </h3>
      <p> {prop}</p>
    </div>
  );
}

Every time "passedProp" changes, our MemoComponent re-renders.

React.memo & state of the component

What if the component wrapped into React.memo HOC has its own state and this state changes?
Is it going to re-render or not?

Now, our MemoComponent has one state variable - "randomNumber" and the button to modify it.

const MemoComponent = React.memo(() => {
  const [randomNumber, setRandomNumber] = useState(Math.random());
  console.info("Child Component - uses memo HOC");

  return (
    <div>
      <h3> Memo Component </h3>
      <p> {randomNumber}</p>
      <button onClick={() => setRandomNumber(Math.random())}>Set random</button>
    </div>
  );
});

Every time we change "randomNumber", our component gonna re-render.

So, if your component has a useState, useContext, or useReducer in its implementation, it will re-render when the state (context) changes.

When to use it ?

data-heavy components that are provided the same props all the time
big size component that has a decent amount of UI elements

Why not use it everywhere ?

Probably you thought about that.

But !!!

Internally React.memo compares props (their previous and new state) so it can decide whether to re-render component or not (if props changed - it should re-render, otherwise not)

And most of the time, computation for this comparison can be even more expensive and take even more time than just re-rendering the component

That is why you should not use React.memo if:

component is cheap to re-render
passed props change often (so there is no meaning to use memo, the component will re-render anyway)
comparison function is expensive to perform

Also, you should not use it as a way to "prevent" a render.
It can lead to bugs!

And the last thing I want to mention is the custom comparison function that can be passed as the second argument.

This function can perform a comparison of previous and new props, and determine whether the component should re-render or not.

Why would we need this?

Consider this example:

In the "App" component we have an object that consists of 1 property and we pass this object to Memo Component.
We do not modify it anywhere.

function App() {
  const [counter, setCounter] = useState(1);
  const complexObject = useState({ qty: 0 });

  return (
    <div className="App">
      <p> {counter}</p>
      <button onClick={() => setCounter(counter + 1)}> Set Counter</button>
      <div className="childComponents">
        <MemoComponent prop={complexObject} />
      </div>
    </div>
  );
}

const MemoComponent = React.memo(() => {
  console.info("Child Component - uses memo HOC");

  return (
    <div>
      <h3> Memo Component </h3>
    </div>
  );
});

But every time we gonna change the state by incrementing "counter", our MemoComponent is re-rendered (despite the fact we use React.memo)

Why is that happening?

When we change the state of the "App" component, we re-create an object, and React thinks that the passed prop has changed and thus forces MemoComponent to re-render.

So, how to fix it?

Easy-peasy.
We just have to pass the function as a second argument that will compare 2 states of props.

const MemoComponent = React.memo(
  () => {
    console.info("Child Component - uses memo HOC");

    return (
      <div>
        <h3> Memo Component </h3>
      </div>
    );
  },
  (previousProps, nextProps) => {
    return previousProps.prop.qty === nextProps.prop.qty;
  }
);

So, as you can see we check whether the "qty" prop of a passed object has changed or not.

If the state of props is different, we have to return false, and this will cause a component to re-render.

Otherwise, the function returns true and we gonna use the previously rendered output.

And that's it, guys.
Now you are ready to use React.memo in your React projects!

I hope that you have learned something new today!
I would appreciate it if you could like this post or leave a comment below!
Also, feel free to follow me on GitHub and Medium!

Adios, mi amigos)