DEV Community: Chetan

🚀 Introducing ✨ Bose Framework - The Swiss Army Knife for Bot Developers 🤖

Chetan — Wed, 24 May 2023 16:40:50 +0000

Bot Development is Tough.

Bot Detectors like Cloudflare are ready to defend websites from our Bots. Configuring Selenium with ChromeOptions to specify the driver path, profile, user agent, and window size is Cumbersome and a nightmare in windows. Debugging Bot Crashes via logs is hard. How do you solve these pain points without sacrificing speed and handy development?

Enter Bose. Bose is the first bot development framework in the Developer Community that is specifically designed to provide the best developer experience for bot developers. Powered by Selenium, it offers a range of features and functionalities to simplify the process of bot development. As far as our knowledge goes, Bose is the first bot development framework of its kind in the town.

Getting Started

Clone Starter Template

git clone https://github.com/omkarcloud/bose-starter my-bose-project

Then change into that directory, install dependencies, and start the project:

cd my-bose-project
python -m pip install -r requirements.txt
python main.py

The first run will take some time as it downloads the chrome driver executable, subsequent runs will be fast.

Core features

Adds Powerful Methods to make working with Selenium a lot easier.
Follows best practices to avoid Bot Detection by Cloudflare and PerimeterX.
Saves the HTML, Screenshot, and the run details for each task run to enable easy debugging.
Utility components to write scraped data as JSON, CSV, and Excel files.
Automatically downloads and initializes the correct Chrome driver.
Fast and Developer friendly.

Usage

Say you want to start scraping a website. If you were using bare Selenium, you would have to handle the imperative tasks of opening and closing the driver like this:

from selenium import webdriver

driver_path = 'path/to/chromedriver'

driver = webdriver.Chrome(executable_path=driver_path)

driver.get('https://www.example.com')

driver.quit()

However, with Bose Framework, you can take a declarative and structured approach. You only need to write the following code, and Bose driver will take care of creating the driver, passing it to the run method of the Task, and closing the driver:

from bose import *

class Task(BaseTask):
    def run(self, driver):
        driver.get('https://www.example.com')

Configuration

In bare Selenium, if you want to configure options such as the profile, user agent, or window size, it requires writing a lot of code, as shown below:

from selenium.webdriver.chrome.options import Options
from selenium import webdriver

driver_path = 'path/to/chromedriver.exe'

options = Options()

profile_path = '1'

options.add_argument(f'--user-data-dir={profile_path}')

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37")'
options.add_argument(f'--user-agent={user_agent}')

window_width = 1200
window_height = 720
options.add_argument(f'--window-size={window_width},{window_height}')

driver = webdriver.Chrome(executable_path=driver_path, options=options)

On the other hand, Bose Framework simplifies these complexities by encapsulating the browser configuration within the BrowserConfig property of the Task, as shown below:

from bose import BaseTask, BrowserConfig, UserAgent, WindowSize

class Task(BaseTask):
    browser_config = BrowserConfig(user_agent=UserAgent.user_agent_106, window_size=WindowSize.window_size_1280_720, profile=1)

Exception handling

Exceptions are common when using Selenium. In bare Selenium, if an exception occurs, the driver automatically closes, leaving you with only logs to debug.

In Bose, when an exception occurs in a scraping task, the browser remains open instead of immediately closing. This allows you to see the live browser state at the moment the exception occurred, which greatly helps in debugging.

Debugging

Web scraping can often be fraught with errors, such as incorrect selectors or pages that fail to load. When debugging with raw Selenium, you may have to sift through logs to identify the issue. Fortunately, Bose makes it simple for you to debug by storing information about each run.

After each run a directory is created in tasks which contains three files, which are listed below:

`task_info.json`

It contains information about the task run such as duration for which the task run, the ip details of task, the user agent, window_size and profile which used to execute the task.

`final.png`

This is the screenshot captured before driver was closed.

`page.html`

This is the html source captured before driver was closed. Very useful to know in case your selectors failed to select elements.

`error.log`

In case your task crashed due to exception we also store error.log which contains the error due to which the task crashed. This is very helful in debugging.

Outputting Data

After performing web scraping, we need to store the data in either JSON or CSV format. Typically, this process involves writing a significant amount of imperative code which looks like this:

import csv
import json

def write_json(data, filename):
    with open(filename, 'w') as fp:
        json.dump(data, fp, indent=4)

def write_csv(data, filename):
    with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
        fieldnames = data[0].keys()  # get the fieldnames from the first dictionary
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writeheader()  # write the header row
        writer.writerows(data)  # write each row of data

data = [
    {
        "text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d",
        "author": "Albert Einstein"
    },
    {
        "text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d",
        "author": "J.K. Rowling"
    }
]

write_json(data, "data.json")
write_csv(data, "data.csv")

Bose simplifies these complexities by encapsulating them in Output Module for reading and writing Data.

To use Output Method, call the write method for the type of file you want to save.

All data will be saved in the output/ folder:

See following Code for Reference

from bose import Output

data = [
    {
        "text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d",
        "author": "Albert Einstein"
    },
    {
        "text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d",
        "author": "J.K. Rowling"
    }
]

Output.write_json(data, "data.json")
Output.write_csv(data, "data.csv")

Undetected Driver

Ultrafunkamsterdam created a ChromeDriver that has excellent support for bypassing all major bot detection systems such as Distil, Datadome, Cloudflare, and others.

Bose recognized the importance of bypassing bot detections and provides in built support for Ultrafunkamsterdam’s Undetected Driver

Using the Undetected Driver in Bose Framework is as simple as passing the use_undetected_driver option to the BrowserConfig, like so:

from bose import BaseTask, BrowserConfig

class Task(BaseTask):
    browser_config = BrowserConfig(use_undetected_driver=True)

LocalStorage

Just like how modern browsers have a local storage module, Bose has also incorporated the same concept in its framework.

You can import the LocalStorage object from Bose to persist data across browser runs, which is extremely useful when scraping large amounts of data.

The data is stored in a file named local_storage.json in the root directory of your project. Here's how you can use it:

from bose import LocalStorage

LocalStorage.set_item("pages", 5)
print(LocalStorage.get_item("pages"))

Boss Driver

The driver you receive in the run method of the Task is an extended version of Selenium that adds powerful methods to make working with Selenium much easier. Some of the popular methods added to the Selenium driver by Bose Framework are:

METHOD	DESCRIPTION
get_by_current_page_referrer(link, wait=None)	simulate a visit that appears as if you arrived at the page by clicking a link. This approach creates a more natural and less detectable browsing behavior.
js_click(element)	enables you to click on an element using JavaScript, bypassing any interceptions(ElementClickInterceptedException) from pop-ups or alerts
get_cookies_and_local_storage_dict()	returns a dictionary containing "cookies" and "local_storage”
add_cookies_and_local_storage_dict(self, site_data)	adds both cookies and local storage data to the current web site
organic_get(link, wait=None)	visits google and then visits the “link” making it less detectable
local_storage	returns an instance of the LocalStorage module for interacting with the browser's local storage in an easy to use manner
save_screenshot(filename=None)	save a screenshot of the current web page to a file in tasks/ directory
short_random_sleep() and long_random_sleep():	sleep for a random amount of time, either between 2 and 4 seconds (short) or between 6 and 9 seconds (long)
get_element_or_* [eg: get_element_or_none, get_element_or_none_by_selector, get_element_by_id, get_element_or_none_by_text_contains,]	find web elements on the page based on different criteria. They return the web element if it exists, or None if it doesn't.
is_in_page(target, wait=None, raise_exception=False)	checks if the browser is in the specified page

Bose is an excellent framework that simplifies the boring parts of Selenium and web scraping.

Wish you best of Luck and Happy Bot Development with Bose Framework!

Learn More

To learn about Bose Bot Development Framework in detail, read the Bose docs at https://www.omkar.cloud/bose/

If Bose Framework helped in Bot Development, please take a moment to star the repository. Your act of starring will help developers in discovering our Repository and contribute towards helping fellow developers in Bot Development. Dhanyawad 🙏! Vande Mataram!

Youtube Collabration?

Chetan — Tue, 23 May 2023 08:40:43 +0000

Are you interested in youtube Collaboration?

🌈Omkar DevTools: A Swiss Army knife for Ninja Developers🚀

Chetan — Mon, 22 May 2023 18:27:55 +0000

Maybe you just know PowerToys. It's a set of utilities for power users to tune and streamline their Windows experience for greater productivity.

DevTools is more or less the same thing, but for developers.

Introduction

Before Omkar DevTools, you must be using a lot of online tools on different sites.

Every time you need to remember the url of the tool, or open Notion to find my notes about the tools and I wasted a lot of times.

Not any more with Omkar DevTools you can access a number of tools in one Dashboard.

At the moment there a lot of tools available inside Omkar DevTools:

JSON to YAML
YAML to JSON
Base64 Encoder
UUID Generator
Secret Key Generator
Hash Generator
Lorem Ipsum Generator
JWT Decoder
Number Base Converter
HTML Encoder and Decoder
URL Encoder and Decoder
GZIP Compressor/Decompressor
Color Blindness Simulator
File Hash Generator
Base64 File Encoder
Case Convertor
Text Counter
Json Duplicate Remover

How to use DevTools

You can easily use Omkar Tools by visiting https://www.omkar.cloud/devtools/

Final Thoughts

Try Omkar Devtools at https://www.omkar.cloud/devtools/

I am curious to hear your thoughts in the Comments about Omkar DevTools.

Dhanyawad 🙏

🌈 A One Person Startup Tech Stack for Ninja Developers - Next.js, Django, Kubernetes, and GCP 🚀

Chetan — Mon, 22 May 2023 16:17:30 +0000

Introduction

In this article, I will share the tech stack I used to build my startup, Omkar. My startup consists of several components, including Frontend, Backend, Blog.

By learning about my tech stack, you can gain insights that will assist you in selecting the tech stack for your startup.

My Application Omkar Cloud

My Application Omkar Cloud is a full stack website with a Blog, developed using technologies like Django, Next.js, and Kubernetes

It is a People Search Engine that you can use, for example, to find people working in a company or CEOs of companies in an industry. Think of it as an advanced version of LinkedIn Search.

Tech Stack of Omkar Cloud

The Most Important Technologies in my Tech Stack are Kubernetes, GCP, Elastic UI, Next.js, GitHub Actions, and Django.

Here is a breakdown of my Tech Stack

Frontend**

For the Frontend I have used Elastic UI, React and Next.js

Elastic UI

I have used many Component Libraries in the past, such as Bootstrap and Material UI. However, based on my experience, I have found the Component Library of Elastic UI to be particularly impressive. Elastic UI is really beautiful Component Library.

Therefore, I have chosen Elastic UI as my UI framework

React

Also, I had learnt two frameworks to create websites which are Svelte and React.

I wanted to use Svelte as it is much more concise than React. However, since the Elastic UI Library was not available for Svelte, I had no choice but to choose React as the only option available to me.

Next.js

Next.js is a framework built on top of React.

I think of Next.js as a framework for a framework 😂.

Next.js simplifies the usage of React and provides additional features. Therefore, instead of using the bare create-react-app setup, I opted to use Next.js as it makes working with React much easier

Backend

For the Backend I used Django powered by SQLite Database

Django

I have learned various backend frameworks such as Express.js and Nest.js, but Django stood out as my favorite.

When working with Django, I found that I could accomplish tasks with significantly fewer lines of code compared to Nest.js or Express.js.

Django's conciseness is similar to Sanskrit Language, in Sanskrit you can convey your thoughts in very fewer words compared to English or Hindi Language.

Hence, I chose Django for its conciseness and simplicity.

SQLite

For the database, I had the option of using either SQLite or PostgreSQL. I opted for SQLite due to several reasons.

Most Importantly for me as an Indian is that it is SQLite is cheaper than PostgreSQL since in SQLite there is no need to purchase a separate server.
It allows developers to start developing faster since PostgreSQL requires spinning up a server, whereas SQLite is file-based
It is easier to view table contents using SQLite Browser application in SQLite compared to PostgreSQL

Although, it can be argued that PostgreSQL is more Scalable but for a Start Up, SQLite does the Job Perfectly. Also, in future you can always migrate to PostgreSQL if necessary.

Blog

To create the blog for my website, I utilized the tailwind-nextjs-starter-blog developed by Timothy Lin. It is a beautiful blogging platform built on Next.js.

Deployment

For deployment, I used Kubernetes, GitHub Actions, and Google Cloud Platform (GCP).

Kubernetes

I wanted to host the frontend, backend, and blog on a single domain, namely 'www.omkar.cloud' at different paths. Additionally, I needed to store the SQLite database file in storage.

Kubernetes provided a convenient method for orchestrating these needs. So, I used Kubernetes.

GitHub Actions

To automate the deployment process, I utilized GitHub Actions. With this setup, I could easily deploy a new version of my application by simply pushing the code to the master branch.

GCP

I chose GCP as the platform to run my entire Kubernetes stack. Google products have a reputation for quality, and GCP proved to be reliable and suitable for my needs. It costs me 2900 INR or $35 per month to run my full Stack.

Other Tools

Google Analytics.

I used Google Analytics to track my website as It was the analytics software I was most familiar with.

G Suite

For creating a professional email address like info@omkar.cloud, I utilized G Suite. I opted for G Suite due to my familiarity with the Gmail interface and my resistance to change.

Google Search Console

I utilized Google Search Console to monitor the search ranking of my website, omkar.cloud, on Google.

NameCheap

I used NameCheap for buying domain name of omkar.cloud. I was satisfied with Name Cheap’s services.

Final Thoughts

In short, I used the following technologies in my tech stack for omkar.cloud:

Frontend:

Elastic UI
React
Next.js

Backend:

Django
SQLite

Blog:

tailwind-nextjs-starter-blog

Deployment:

Kubernetes
GitHub Actions
Google Cloud Platform (GCP)

Other Tools:

Google Analytics
G Suite
Google Search Console
NameCheap

Overall, I am very satisfied with the tech stack of omkar.cloud.

If you are creating a startup and considering the tech stack to use, based on my experience, I can confidently say that if you base your tech stack on mine, you will have a solid technological foundation for your startup.

I am curious to hear if you have any questions regarding the tech stack, so please feel free to ask in the comments.

Dhanyawad 🙏

🔥💥🚀 10 Missing Selenium Methods for Ninja Developers!😱💥🔥

Chetan — Mon, 22 May 2023 11:41:12 +0000

Introduction

Based on my experience, I have created multiple utilities to simplify Selenium for bot developers. First, let me describe these utilities to you, and then I will explain how to use them effectively.

Methods

1) get_by_current_page_referrer(link, wait=None):

An alternative to driver.get is the get_by_current_page_referrer(link, wait=None) utility. When using driver.get, the document.referrer property remains empty, indicating that you directly entered the page URL in the search bar, which can raise suspicion for bot detection systems.

By employing get_by_current_page_referrer(link, wait=None), you can simulate a visit that appears as if you arrived at the page by clicking a link. This approach creates a more natural and less detectable browsing behavior.

In general, when navigating to an internal page of a website, it is recommended to replace driver.get with get_by_current_page_referrer. Additionally, you have the option to specify the amount of time to wait before navigating, using the optional wait parameter.

driver.get_by_current_page_referrer("https://example.com")

2) js_click(element):

While clicking elements with Selenium, elements can be intercepted by pop-ups, alerts, or other elements, leading to the raising of an ElementClickInterceptedException error.

To handle such situations, you can utilize the js_click method. This method enables you to click on an element using JavaScript, bypassing any interceptions from pop-ups or alerts.

By employing the js_click method, you can ensure that the click operation is executed smoothly without being intercepted.

button = driver.get_element_or_none_by_selector(".button")
driver.js_click(button)

3) get_cookies_and_local_storage_dict():

This method returns a dictionary containing two keys, "cookies" and "local_storage", each of which contains a dictionary of the cookies and local storage. You can use them to persist session by storing them in a JSON file.

site_data = driver.get_cookies_and_local_storage_dict()

4) add_cookies_and_local_storage_dict(self, site_data):

This method adds both cookies and local storage data to the current web site.

site_data = {
    "cookies": {"cookie1": "value1", "cookie2": "value2"},
    "local_storage": {"name": "John", "age": 30}
}
driver.add_cookies_and_local_storage_dict(site_data)

5) organic_get link, wait=None):

This method follows a two-step process: it first loads the Google homepage and then navigates to the specified link. This approach closely resembles the way people typically visit websites, resulting in more humane behavior and reduces chances of bot being detected.

driver.organic_get("https://example.com")

6) local_storage

This property returns an instance of the LocalStorage class from the bose.drivers.local_storage module. This class is used for interacting with the browser's local storage in an easy to use manner.

local_storage = driver.local_storage

# Set an item in the Browser's Local Storage
local_storage.set_item('username', 'johndoe')

# Retrieve an item from the Browser's Local Storage
username = local_storage.get_item('username')

7) save_screenshot(filename=None)

Use this method to save a screenshot of the current web page to a file in tasks/ directory. The filename of the screenshot is generated based on the current date and time, unless a custom filename is provided.

driver.save_screenshot()

8) short_random_sleep() and long_random_sleep():

These methods sleep for a random amount of time, either between 2 and 4 seconds (short) or between 6 and 9 seconds (long). You can use them like this:

driver.short_random_sleep()
driver.long_random_sleep()

9) get_element_or_none(xpath, wait=None), get_element_or_none_by_selector(selector, wait=None), get_element_by_id(id, wait=None), get_element_or_none_by_text_contains(text, wait=None), get_element_or_none_by_text(text, wait=None), get_element_or_none_by_name(selector, wait=None):

These methods find web elements on the page based on different criteria. They return the web element if it exists, or None if it doesn't. You can also pass number of seconds to wait for element to appear. You can use them like this

# find an element by xpath
element = driver.get_element_or_none("//div[@class='example']", 4)

# find an element by CSS selector
element = driver.get_element_or_none_by_selector(".example-class", 4)

# find an element by ID
element = driver.get_element_by_id("example-id", 4)

# find an element by text
element = driver.get_element_or_none_by_text("Example text", 4)

# find an element by partial text
element = driver.get_element_or_none_by_text_contains("Example", 4)

# find an element with attribute name = "email"
element = driver.get_element_or_none_by_name("email", 4)

10) is_in_page(target, wait=None, raise_exception=False)

This method checks if the browser is in the specified page. It will keep checking the URL for a specified amount of time (if wait is provided) and return False if the target is not found. If raise_exception is True, it will raise an exception if the page is not found.

is_in_page = driver.is_in_page('example.com', wait=10, raise_exception=True)

How to use these utilities?

The easiest way to use these utilities is to utilize the Bose Framework, which is the first framework designed to simplify Bot Development for Developers.

Bose Framework automatically incorporates these utility methods into the Selenium Driver. You can learn how to use the Bose Framework by following the tutorial at https://www.omkar.cloud/bose/docs/tutorial/.

🚀 🤖💻🔍 How to scrape g2 using Python, Selenium and Bose Framework 🅶2️⃣🐍🖥️

Chetan — Sun, 21 May 2023 15:07:45 +0000

Introduction

In this article, you will learn how to scrape g2.com using Bose Framework.

Also, Scraping g2.com is an excellent way to do competitor analysis.

Bose Framework, is a Selenium based Bot Development Framework that provides a comprehensive set of tools and functionalities specifically aimed at making the Bot Development Process easy for Developers.

To make it easy to scrape g2.com, I have prepared a script that you can use to scrape g2 effectively. This article will walk you through the steps of utilizing the script.

Installation

Clone Starter Template

git clone https://github.com/omkarcloud/g2-scraper
cd g2-scraper

Install dependencies

python -m pip install -r requirements.txt

Usage

In extract_product_links.py specify your Task.product_url
Run Project

python main.py

The script will start running and output progress updates to the console. When the scraper is complete, it will generate a JSON file named pending.json in the output directory. The JSON file will contain the product links.

Once the bot is detected by Cloudflare, the script will recognize it and prompt you to press the "Enter" key in the console once you have successfully solved the Cloudflare captcha.

Additionaly, you don't have to configure the Selenium driver as it will automatically download the appropriate driver based on your Chrome browser version.

In main.py change task variable to src.extract_product_links
Rerun Project

python main.py

The products will be extracted and stored in the output/finished.csv and output/finished.json file after scraping.

🤔🌐🗺️ How to scrape google maps using Python, Selenium and Bose Framework 🐍🔍🌍

Chetan — Sun, 21 May 2023 12:23:48 +0000

Introduction

In today's digital era, businesses are constantly seeking innovative ways to generate leads and gain a competitive edge. One incredibly powerful tool that has become indispensable for lead generation is Google Maps.

With its vast database of businesses and geolocation information, scraping Google Maps can provide valuable insights and opportunities for various industries. It allows businesses to extract information such as business names, contact details, addresses, customer reviews, and more. This data can serve as a goldmine for web developers and marketers in their quest for B2B leads.

To Scrape Google Maps we will use Bose Framework, a Selenium based Bot Development Framework that provides a comprehensive set of tools and functionalities specifically aimed at making the Bot Development Process easy for Developers.

Let's dive into the fascinating world of lead generation and explore how web developers can leverage the Bose Framework and Google Maps scraping to discover B2B leads.

How to scrape google maps

I have written a Python script that allows you to scrape information from Google Maps, including business names, addresses, phone numbers, websites, ratings, and reviews. The script can be configured to search for specific queries and can scrape either the first page of results or all pages of results.

Let's get started with Installation

Installation

Clone Starter Template ```bash

git clone https://github.com/omkarcloud/google-maps-scraper
cd google-maps-scraper

2. Install dependencies
```bash


python -m pip install -r requirements.txt

Run Project ```bash

python main.py


The script will start running and output progress updates to the console. When the scraper is complete, it will generate a CSV file named `finished.csv` in the `output` directory. The CSV file will contain the business name, address, phone number, website, rating, and review for each result.

Additionaly, you don't have to configure the Selenium driver as it will automatically download the appropriate driver based on your Chrome browser version.

## Configuration

- To specify the Google search queries to be used in the scraper, open the `src/scraper.py` file in your preferred text editor and update the `Task.queries` list with your desired queries.

- To specify whether to scrape the first page of Google Maps results or all pages of results, open the `src/scraper.py` file and set the `Task.GET_FIRST_PAGE` variable to `True` or `False` as appropriate.

⭐🤖🔑 9 Ninja Tips to prevent Bot from getting Detected like Grey Ninja 🕵️‍♂️🔍🛡️

Chetan — Sat, 20 May 2023 16:33:14 +0000

Introduction

Imagine this: You've writing a Selenium script to automate a website, only to be blocked by detection services like Cloudflare and PerimeterX.

In this Article, I will share 9 Brilliant Tips to prevent Cloudflare and PerimeterX from detecting your Bots.

Within this article, I will also introduce you to Bose Framework, a framework that I created using my bot development skills to help bot developers in creating unstoppable Bots. Think of it like Swiss Army Knife for Bot Development.

I have concentrated all my Knowledge of Bot Development into a Single Easy to use Framework for the benefit of all Bot Developers.

Now, Let us learn the Tips to avoid bot detection from Cloudflare and PerimeterX like a Real Ninja.

Tips

1. User-Agent and Window Size Rotation

Bot Detection systems analyze the user agent and window size to identify bots.

For instance, making an abnormal number of requests (e.g., 1000 page visits) using a single user agent increases the chances of being flagged as a bot.

Also, using made up user agents that are not in usage also increases the chances of being flagged as a bot.

So, it is best to use commonly used User Agents like Chrome 104 and Chrome 106 on a random basis.

In Bose Framework, the following code snippet demonstrates how to achieve random rotation of user agents and window sizes with each run:

from bose import *

class Task(BaseTask):
    browser_config = BrowserConfig(user_agent=UserAgent.RANDOM, window_size=WindowSize.RANDOM)

Additionally, if you are utilizing the Bose Framework instead of plain Selenium, the user agent will be automatically rotated on each run by default.

2. Change your IPs

To evade detection, changing your IP address is another crucial aspect. Websites often track IP addresses to identify and block suspicious activity.

If you have been detected by services like Cloudflare or PerimeterX, you will need to change your IP address to continue automation.

There are both free and paid methods to change your IP address:

Free Methods

Enable and disable airplane mode

In this method, you can utilize the connection between your PC and a mobile hotspot to access the internet via a SIM card and then enable and disable airplane mode on your mobile device to get a new IP address.

This method is quite fast and usually takes 10 seconds to acquire a new IP.

Here's how you can enable and disable airplane mode to change your IP address:

Connect your PC to a mobile hotspot.
On your mobile device, turn airplane mode on and off
Now, Turn the Hotspot On Again.

On and Off WIFI Router

For devices connected to a Wi-Fi network, turning off the router and then turning it back on will change the IP address. Although It is slow methods as Wifi Router will take 2-3 minutes to start.

Paid Method

Paid services like Oxylab and IPRoyal provide a pool of residential IP addresses. However, they can be quite expensive, with prices averaging around $15 per GB for residential IPs. If you opt for this method and are scraping large amounts of data, consider disabling the loading of CSS and images to reduce costs.

In general, the free method of using airplane mode works well for most use cases. It is not only free but also faster than using a proxy

3. Humane Page Flow

To avoid suspicion, replicate organic user behavior by engaging with the website as a human would. In general you should follow these recomendation.

Access pages via search engines

Instead of directly visiting a page, navigate to it through a search engine like Google. This emulates how users typically discover websites by searching on a search engine. The Bose Framework provides a special method called organic_get in the Selenium driver, which achieves this behavior. You can use it as follows:

driver.organic_get("https://example.com")

Navigate using internal links

Browsers can detect the page you came from by checking the document.referrer property. When visiting multiple pages within a site, utilize the get_by_current_page_referrer method from the Bose driver. This JavaScript-based method simulates internal navigation by clicking a link, making it appear more natural:

driver.get_by_current_page_referrer("https://example.com")

4. Use Random Waits and Slow the Bot

Adding random wait times and slowing bot between actions helps mimic human interaction patterns and evade bot detection.

The Bose Framework offers short_random_sleep() and long_random_sleep() methods that pause the bot for a random duration.

Here is how you can use them:

driver.short_random_sleep()
driver.long_random_sleep()

5. Use Captcha Solving Services

Services like Cloudflare may present CAPTCHA challenges. To overcome this obstacle, you can use CAPTCHA solving services like 2captcha, which solves the CAPTCHA challenges.

6. Use Ultrafunkamsterdam’s ChromeDriver

Ultrafunkamsterdam's ChromeDriver, developed by Leon from the Netherlands, is an exceptional tool that enhances the stealth of Selenium Driver . It applies patches making it harder for detection services like Cloudflare and PerimeterX to detect the Bot.

The Bose Framework seamlessly integrates with Ultrafunkamsterdam’s ChromeDriver. To use it, specify use_undetected_driver=True in the browser_config:

class Task(BaseTask):
    browser_config = BrowserConfig(use_undetected_driver=True)

7. Do not use a Headless Browser

Avoid using headless browsers, as bot detection systems like PerimeterX can detect them by checking for specific APIs.

For example, the following code in JavaScript determines whether the browser is headless. In the case of headless Chrome, the test will log "Headless Chrome"

navigator.permissions
  .query({ name: "notifications" })
  .then(function (permissionStatus) {
    if (
      Notification.permission === "denied" &&
      permissionStatus.state === "prompt"
    ) {
      console.log("Headless Chrome")
    } else {
      console.log("Not Headless Chrome")
    }
  })

8. Maintain Consistency in User Agent, Window Size, and IP Address

If you are maintaining sessions by using a profile in Selenium, then make sure that the User Agent and Window Size remains same for the Profile.

Sudden and frequent changes in these parameters like one time you are using Chrome 106 and then Chrome 98 and then Chrome 102 can raise red flags and lead to detection.

In the Bose Framework, if you specify a profile in a task, it will automatically retain the same user agent and window size on each run using that profile. Here's an example of how to specify a profile in the Boss Task:

class Task(BaseTask):
    # The User Agent and Window Size will remain same on each run with profile 1.
    browser_config = BrowserConfig(profile=1)

Also, When using a proxy, it is equally important to maintain consistency in the IP address location. Avoid abrupt changes from one country to another, such as switching from Bharat (India) to Russia and then to Israel.

9. Use Bose Framework

Whenever you are developing any Selenium Script, you should definetly use Bose Framework.

Just like a Thunder Stone evolves Pikachu into Raichu in the Pokémon world, the Bose Framework transforms a normal bot developer into a mega bot developer.

Read the Bose Framework Tutorial at https://www.omkar.cloud/bose/docs/tutorial/ to create powerful bots.

💻🎯 Ninja Kubernetes Cheat Sheet for Ninja Developers 📜

Chetan — Thu, 18 May 2023 13:58:35 +0000

A small request

I'm trying to reach 1K GitHub stars for "Bose" - an open source bot development framework I just launched for easily creating advanced selenium based bots.
Can you help us out by starring the GitHub repository? It would help us a lot! Thank you! https://github.com/omkarcloud/bose

# Ger logs of Pods in a Deployment. This is the recommended way to fetch logs, as the deployment name remains the same even if pods change upon creation.
kubectl logs deploy/<deployment-name>

# View the latest logs from the end. This can be useful when there are a large number of logs generated by a pod.
kubectl logs deploy/<deployment-name> --tail=2000

# Access a Pod within a Deployment using an interactive bash shell. This is the recommended method as the deployment name remains the same, unlike the pods that change upon creation.
kubectl exec -it deploy/<deployment-name> -- bash

# Access a Pod within a Deployment using an interactive shell. Use this if the previous method fails, as the pod does not support bash.
kubectl exec -it deploy/<deployment-name> -- sh

# Apply a folder or file containing Kubernetes manifests.
kubectl apply -f <folder-name-or-filename>

# Delete a folder or file containing Kubernetes manifests.
kubectl delete -f <folder-name-or-filename>

# Delete a deployment.
kubectl delete deployment <deployment-name>

# Forcefully delete a deployment without waiting for graceful termination.
kubectl delete deployment <deployment-name> --grace-period=0 --force

# Display the logs of a specific pod.
kubectl logs <pod-name>

# Stream and display the logs of a specific pod.
kubectl logs <pod-name> --follow

# Delete all jobs.
kubectl delete jobs `kubectl get jobs -o custom-columns=:.name`

# Delete pods based on a selector.
kubectl delete pods -l <pod-selector-key>=<pod-selector-value>

Pods

# List all pods in the current namespace.
kubectl get pods

# Get detailed information about a pod.
kubectl describe pod <pod-name>

# Create a new pod.
kubectl create pod <pod-name> 

# Delete a pod.
kubectl delete pod <pod-name>

Nodes

# List all nodes in the cluster.
kubectl get nodes 

# Get detailed information about a node.
kubectl describe node <node-name> 

# Create a new node
kubectl create node <node-name> 

# Delete a node
kubectl delete node <node-name>

Services

# List all services in the cluster.
kubectl get services 

# Get detailed information about a service.
kubectl describe service <service-name> 

# Create a new service.
kubectl create service <service-name> 

# Delete a service.
kubectl delete service <service-name>

Deployments

# List all deployments in the cluster.
kubectl get deployments 

# Get detailed information about a deployment.
kubectl describe deployment <deployment-name> 

# Create a new deployment.
kubectl create deployment <deployment-name> 

# Delete a deployment.
kubectl delete deployment <deployment-name>

ReplicaSets

# List all replica sets in the cluster.
kubectl get replicasets 

# Get detailed information about a replica set.
kubectl describe replicaset <replicaset-name> 

# Create a new replica set.
kubectl create replicaset <replicaset-name> 

# Delete a replica set.
kubectl delete replicaset <replicaset-name>

StatefulSets

# List all stateful sets in the cluster.
kubectl get statefulsets 

# Get detailed information about a stateful set.
kubectl describe statefulset <statefulset-name> 

# Create a new stateful set.
kubectl create statefulset <statefulset-name> 

# Delete a stateful set.
kubectl delete statefulset <statefulset-name>

Jobs

# List all jobs in the cluster.
kubectl get jobs 

# Get detailed information about a job.
kubectl describe job <job-name>

# Create a new job.
kubectl create job <job-name> 

# Delete a job.
kubectl delete job <job-name>

CronJobs

# List all cron jobs in the cluster.
kubectl get cronjobs 

# Get detailed information about a cron job.
kubectl describe cronjob <cronjob-name> 

# Create a new cron job.
kubectl create cronjob <cronjob-name> 

# Delete a cron job.
kubectl delete cronjob <cronjob-name>

ConfigMaps

# List all config maps in the cluster.
kubectl get configmaps 

# Get detailed information about a config map.
kubectl describe configmap <configmap-name> 

# Create a new config map.
kubectl create configmap <configmap-name> 

# Delete a config map.
kubectl delete configmap <configmap-name>

Secrets

# List all secrets in the cluster.
kubectl get secrets 

# Get detailed information about a secret.
kubectl describe secret <secret-name> 

# Create a new secret.
kubectl create secret <secret-name> 

# Delete a secret.
kubectl delete secret <secret-name>

Ingresses

# List all ingresses in the cluster.
kubectl get ingresses 

# Get detailed information about an ingress.
kubectl describe ingress <ingress-name>

# Create a new ingress.
kubectl create ingress <ingress-name> 

# Delete an ingress.
kubectl delete ingress <ingress-name>

PersistentVolumes

# List all persistent volumes in the cluster.
kubectl get pv 

# Get detailed information about a persistent volume.
kubectl describe pv <persistentvolume-name> 

# Create a new persistent volume.
kubectl create pv <persistentvolume-name> 

# Delete a persistent volume.
kubectl delete pv <persistentvolume-name>

PersistentVolumesClaims

# List all persistent volumes in the cluster.
kubectl get pvc

# Get detailed information about a persistent volume.
kubectl describe pvc <persistentvolumesclaim-name> 

# Create a new persistent volume.
kubectl create pvc <persistentvolumesclaim-name> 

# Delete a persistent volume.
kubectl delete pvc <persistentvolumesclaim-name>

Also, this article is licensed under MIT, so you are free to use it as you wish.

If you are a YouTuber, I encourage you to create a video of this article on your channel. By doing so, you can potentially help thousands of developers solve their deployment problems and earn great amounts of punya. Additionally, you can send me your Video URL and I will feature your video at the top of the article, boosting its views.

Some of my brilliant Creations, that you don’t want to miss out are:

Bose Framework: An Open Source Framework that is Swiss Army Knife for Selenium. Read the docs at https://www.omkar.cloud/bose/
Free Tools: Simplify your daily tasks with the Free Omkar Tools, an all-in-one dashboard that offers a range of free utilities. Whether it's formatting JSON, comparing text, or testing RegExp, these tools have got you covered. Say goodbye to unreliable websites and use it at https://www.omkar.cloud/tools/.
Omkar Cloud: My Samurai Lead Generation Software allows you to access over 200M+ LinkedIn leads, enabling you to win clients. It's like Sales Navigator with superpowers. You can use it at https://www.omkar.cloud/.
Omkar Cloud Datasets: Enhance your analysis with meticulously scraped datasets from Stack Overflow, G2, and many more reputable sources. These fresh datasets are readily available for your data analysis at https://www.omkar.cloud/datasets/.
Kubernetes Consultation: Take your DevOps practices to the next level with our team of Ninja Developers. Through our Kubernetes Consultation service, we can help you turbocharge your DevOps processes and enable you to move at the speed of light. Whatsapp us for a consultation at https://www.omkar.cloud/l/whatsapp.

🌍🚀🎯 From Localhost to Cloud ☁️: Next.js, Django, SSL 🔒, GitHub Actions 🚀, DNS| Ultimate Website Deployment Tutorial🌟🔥✨

Chetan — Wed, 17 May 2023 16:59:19 +0000

A small request

Introduction

Picture this: 📸 You have developed a stunning website, and now it's time to make it accessible to the world. But the deployment process seems daunting, right? 😰

Fear not, because Kubernetes is here to save the day. 🦸‍♂️ With its powerful features and intelligent container management, Kubernetes takes the complexity out of website deployment.

In this tutorial, we will demystify Kubernetes and guide you through the process of deploying your website effortlessly. Get ready to unlock the magic of Kubernetes and witness your website thrive in realms of the Cloud. ✨🌐

What we will do

We will guide you through the process of deploying your website from running locally on your localhost to the Cloud.

The website will be a full stack website with frontend and backend with a SQLite database served over SSL.

We will also make a CI Pipeline to deploy code on commit to master Branch.

To accomplish this, we will perform the following steps:

Dockerize the Next.js frontend and Django backend.
Write Kubernetes manifests for frontend and backend servers, including a Persistent Volume for the Django SQLite database and obtaining an SSL certificate using Cert Manager.
Create a Kubernetes cluster in GCP.
Configure the domain's DNS.
Write GitHub Actions YAML for building and deploying applications to GCP.
Push code to GitHub for deployment.

This tutorial involves several technologies, and if you encounter any unfamiliar concepts, don't worry. As you progress through the tutorial, you will gain a better understanding of each technology and its role in the deployment process.

Prerequisites

To successfully complete this tutorial, you will need:

A GCP Account
A Domain Name

If you haven't purchased a domain name yet, I recommend using Namecheap as I have personally used them and had a good experience.

I highly recommend to read through the tutorial 2-3 times before proceeding with the deployment on your own.

Dockerization

For this tutorial, a repository has been created consisting of two folders: frontend and backend.

The frontend folder contains the code for the Next.js-based application, while the backend folder contains the code for the Django-based application.

To begin, you will need to clone the starter code:



git clone https://github.com/omkarcloud/kubernetes-website-deployment-tutorial-starter

Open the project in Visual Studio Code by running:



code kubernetes-website-deployment-tutorial-starter/

Dockerizing Frontend

To Dockerize the frontend code built with Next.js, create a Dockerfile in the frontend folder with the following contents:

frontend/Dockerfile



FROM node:16-alpine

COPY package.json .
RUN rm -rf node_modules/ .next/* && npm install --legacy-peer-deps
COPY . .

RUN npm run build

CMD ["npm", "run", "start"]

Additionally, create a .dockerignore file in the frontend folder with the same contents as .gitignore by running following command:



cp frontend/.gitignore frontend/.dockerignore

Dockerizing Backend

Now, let's Dockerize the Django-based backend code by creating a Dockerfile in the backend folder with the following contents:

backend/Dockerfile



FROM python:3.9
ENV PYTHONBUFFERED 1

COPY requirements.txt .
RUN python -m pip install -r requirements.txt

RUN mkdir app
WORKDIR /app
COPY . /app

CMD gunicorn --workers 3 -b 0.0.0.0:8000 config.wsgi

Next, create a .dockerignore file in the backend folder with the same contents as .gitignore by running following command:



cp backend/.gitignore backend/.dockerignore

Creating PVC for Backend

To ensure persistence of our SQLite database file used in the Django application, let's create a Persistent Volume Claim (PVC) by creating pvc.yaml file at k8s/volumes/ with the following contents, which provisions 4GB of storage in GCP:

k8s/volumes/pvc.yaml



apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
  storageClassName: standard-rwo

Creating Service for Backend and Frontend

To make the backend and frontend pods accessible within the cluster, create the following manifest files:

Backend Deployment (k8s/app/backend-depl.yaml): ```yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-depl
spec:
replicas: 1
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
volumes:
- name: app-data
persistentVolumeClaim:
claimName: csi-pvc
containers:
- name: backend
image: placeholder/backend:1.0.5
volumeMounts:
- mountPath: /app-data

name: app-data

apiVersion: v1
kind: Service
metadata:
name: backend-srv
spec:
selector:
app: backend
ports:
- name: backend
protocol: TCP
port: 8000
targetPort: 8000


2.  Frontend Deployment (`k8s/app/frontend-depl.yaml`):
```yaml


apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - name: frontend
          resources:
            requests:
              ephemeral-storage: "800Mi"
            limits:
              ephemeral-storage: "800Mi"
          image: placeholder/frontend:1.0.4
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-srv
spec:
  selector:
    app: frontend
  ports:
    - name: frontend
      protocol: TCP
      port: 3000
      targetPort: 3000

Creating Load Balancer

To expose our pods via services, we need to create an Ingress resource. Create a file named ingress.yaml in k8s/app/ with the following contents, replacing "www.your-domain.com" with your domain name:

k8s/app/ingress.yaml



apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/proxy-body-size: 80m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
spec:
  rules:
    - host: "www.your-domain.com"
      http:
        paths:
          - path: /backend
            pathType: Prefix
            backend:
              service:
                name: backend-srv
                port:
                  number: 8000
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-srv
                port:
                  number: 3000

  tls:
    - hosts:
        - "www.your-domain.com"
      secretName: ssl-certificate

Getting SSL Certificate

To obtain an SSL certificate for your domain, you can create a YAML file named certificates.yaml in the k8s/certificates directory.

Replace www.your-domain.com with your actual domain name and optionally replace youremail@gmail.com with your email address to receive SSL-related emails from Let's Encrypt (Our SSL Provider).

k8s/certificates/certificates.yaml



apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: youremail@gmail.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx
        selector:
          dnsNames:
            - "www.your-domain.com"

Create Github Actions

To automate the deployment process of your application on each commit to the master branch, you can utilize GitHub Actions.

The following workflow defines the necessary steps to authenticate with Google Cloud Platform (GCP), build Docker images, push them to Google Container Registry, and deploy the application to Google Kubernetes Engine (GKE) cluster.

Please note that the deployment includes deleting the previous backend deployment, which will result in temporary downtime of 4 to 5 minutes whenever you deploy code by pushing to Github.

We also utilize various Google Cloud Platform (GCP) environment variables for the deployment, which we will store securely in the secrets of our GitHub repository later.

To use this workflow create a file called deploy.yaml in the .github/workflows directory. Here is the content for the deploy.yaml file:

.github/workflows/deploy.yaml



name: Build and Deploy to GKE

on:
  push:
    branches:
      - master

env:
  PROJECT_ID: ${{ secrets.GKE_PROJECT }}
  GKE_CLUSTER: my-application-cluster
  GKE_PROJECT: ${{ secrets.GKE_PROJECT }}
  GKE_CLUSTER_NAME: my-application-cluster
  GKE_ZONE: asia-south1-a
  GKE_EMAIL: ${{ secrets.GKE_EMAIL }}
  GKE_KEY: ${{ secrets.GKE_KEY }}
  GITHUB_SHA: ${{ github.sha }}
  ACTIONS_ALLOW_UNSECURE_COMMANDS: "true"
  USE_GKE_GCLOUD_AUTH_PLUGIN: "True"

jobs:
  setup-build-publish-deploy:
    name: Setup, Build, Publish, and Deploy
    runs-on: ubuntu-latest
    environment: production
    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Set Images as enviroment variables
        run: |
          echo "FRONTEND=$(echo "gcr.io/""$GKE_PROJECT""/frontend:""$GITHUB_SHA")" >> $GITHUB_ENV
          echo "BACKEND=$(echo "gcr.io/""$GKE_PROJECT""/backend:""$GITHUB_SHA")" >> $GITHUB_ENV

      - name: update images
        run: |
          FRONTEND_ESCAPE=$(printf '%s\n' "$FRONTEND" | sed -e 's/[\/&]/\\&/g')
          sed -i -e 's/placeholder\/frontend:1.0.4/'"$FRONTEND_ESCAPE"'/g' frontend-depl.yaml
          cat frontend-depl.yaml
          BACKEND_ESCAPE=$(printf '%s\n' "$BACKEND" | sed -e 's/[\/&]/\\&/g')
          sed -i -e 's/placeholder\/backend:1.0.5/'"$BACKEND_ESCAPE"'/g' backend-depl.yaml
          cat backend-depl.yaml
        working-directory: k8s/app

      - id: 'auth'
        uses: 'google-github-actions/auth@v1'
        with:
          credentials_json: '${{ secrets.GCP_CREDENTIALS }}'

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v1

      - name: Install Gke
        run: |
          gcloud components install gke-gcloud-auth-plugin

      - name: Cluster Login
        run: |
          gcloud container clusters get-credentials $GKE_CLUSTER_NAME --zone $GKE_ZONE --project $GKE_PROJECT

      - name: Configure Docker
        run: gcloud --quiet auth configure-docker

      - run: docker build --tag "$FRONTEND" .
        working-directory: frontend

      - run: docker build --tag "$BACKEND" .
        working-directory: backend

      - name: Push Images
        run: |
          docker push "$FRONTEND"
          docker push "$BACKEND"

      - name: Deploy
        run: |
          kubectl delete deployment backend-depl || true
          kubectl apply --recursive -f k8s/
          deploy=$(kubectl get deploy -o name) &&  for i in $deploy; do kubectl rollout status $i -w --timeout=30s; done || true
          kubectl get pods

GCP Setup

Now, we will set up the GCP Cluster.
I will tell you when there you need to change some variables. Please only modify the variables I tell you and keep the other variables untouched, such as the cluster name and zone etc.

As they are hardcoded and will be used multiple times in the commands of this Tutorial. Once you have become familiar with the tutorial, feel free to rename them according to your preference.

Enable the following APIs for your Project to create a GKE cluster: ```bash

Artifact Registry: https://console.cloud.google.com/marketplace/product/google/artifactregistry.googleapis.com
Kubernetes Engine: https://console.cloud.google.com/marketplace/product/google/container.googleapis.com
Compute Engine: https://console.cloud.google.com/marketplace/product/google-cloud-platform/compute-engine


![api-gcp.png](https://www.omkar.cloud/blog/static/blog/from-localhost-to-cloud-next-js-django-ssl-github-actions-dns-configuration-the-ultimate-website-deployment-tutorial-with-kubernetes/api-gcp.png)

2. In the sidebar, click on the "Kubernetes Engine" button.

![gke.png](https://www.omkar.cloud/blog/static/blog/from-localhost-to-cloud-next-js-django-ssl-github-actions-dns-configuration-the-ultimate-website-deployment-tutorial-with-kubernetes/gke.png)

3. Click on the "Create" button to create a new cluster. Configure the "Standard Mode" Cluster, as it is more cost-effective.

![choose.png](https://www.omkar.cloud/blog/static/blog/from-localhost-to-cloud-next-js-django-ssl-github-actions-dns-configuration-the-ultimate-website-deployment-tutorial-with-kubernetes/choose.png)

4. Provide the following details for the cluster:

```bash


Cluster Name: my-application-cluster
Location Type: zonal
Zone: asia-south1-a

In the sidebar, select "default-pool" and change the number of nodes to 1 instead of 3 to save costs.

In the sidebar, select "Nodes" and use the following settings to save costs:



Machine Type: e2-small 
Boot Disk Size: 20 GB

The estimated cost should be around $90. If this is your first cluster in GCP, the cost will be charged around $27 after 1 Month discounting $73 discount for the first cluster.

Click the "Create" button, and the cluster will be created in approximately 5-6 minutes.
Open the GCP Shell by clicking the shell icon in the top-right corner.

Authenticate in your cluster by running the following command. Replace "clear-world-31478" with your GCP project name.



gcloud container clusters get-credentials my-application-cluster --zone asia-south1-a --project clear-world-31478

Create an IP address by running the following command: ```bash

gcloud compute addresses create my-application-cluster-ip --region asia-south1


11. Note the created IP address by running the command:

```bash


gcloud compute addresses describe my-application-cluster-ip --region asia-south1

In your Domain Name Service (DNS) provider, such as Namecheap, add the following records. Replace "31.91.11.253" with the IP address obtained in the previous step.



A @ 31.91.11.253
A www 31.91.11.253

Ensure that the TTL (Time to Live) is set to the lowest possible value, such as 1 minute, to allow for fast DNS propagation.

If you are using Namecheap, select the 1-minute TTL option, which will take 1 minute to propogate DNS Changes.

Execute the following commands to install ingress-nginx and cert-manager which are needed do load balancing get the SSL certificate. In the following commands, replace 31.91.11.253 with the IP address you created earlier.



helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx-chart ingress-nginx/ingress-nginx --set controller.service.loadBalancerIP=31.91.11.253 --set controller.service.externalTrafficPolicy=Local

helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.8.0

Github Setup

This is the final step where we will configure GitHub Secrets for our deployment workflow and push the code.

Create a Github Repository
Create a service account JSON file by visiting https://console.cloud.google.com/iam-admin/serviceaccounts. The service account will have full access to all resources (Owner) in Google Cloud (GCP). Keep the JSON file secure, as it will be required for deploying to Google Kubernetes Engine (GKE). Save the file as account.json.

Create Github secrets which are used in Workflow by going to your Repository GitHub Settings > Secrets > Actions and creating three secrets with the following names and values: ```

GKE_PROJECT="clear-world-31478"
GKE_EMAIL="owner-709@project.iam.gserviceaccount.com"
GKE_KEY="JodHRWCzov...L3D3dy5n"


Here,
  - `GKE_PROJECT` is your project name.
  - `GKE_EMAIL` is email located in the `account.json` file.
  - `GKE_KEY` is your service account encoded as base64. You can obtain it by running `cat account.json | base64`.

4. Now, push the code to the repository. Once pushed, the site will be ready 4-5 minutes after the GitHub Action is completed. It takes 4-5 minutes as it will obtain SSL certificates etc.

You can view status of your actions by visiting Action Tabs of your Github Repository.

After Action is Completed, you can access the website at your domain.

## Delete the Resources

Remember to delete the GCP resources to avoid incurring charges. Follow these steps to delete the GCP resources:

1. Delete the created IP address by running the following command. In this command, replace "clear-world-31478" with your GCP project name.

gcloud compute addresses delete my-application-cluster-ip --region asia-south1 --project clear-world-31478


2. Use the GKE Dashboard to delete the cluster using the graphical user interface.



## FAQ

**Q: How can I connect PostgreSQL to Django in Kubernetes?**

A: If you are just starting out, I highly recommend using SQLite to save both money and development time. However, if you prefer to use PostgreSQL, you can follow the tutorial at https://cloud.google.com/python/django/kubernetes-engine.


**Q: Where is the final Code?**

A: You can find the final code at https://github.com/omkarcloud/kubernetes-website-deployment-tutorial-finished.


## Final Words

Congratulations on successfully deploying your website! You have accomplished website deployment with SSL, DNS, GitHub Actions, etc.

If you have any questions, feel free to ask them in the comments.

--- 

Also, this article is licensed under MIT, so you are free to use it as you wish.

If you are a YouTuber, I encourage you to create a video of this article on your channel. By doing so, you can potentially help thousands of developers solve their deployment problems and earn great amounts of punya. Additionally, you can send me your Video URL and I will feature your video at the top of the article, boosting its views.

---
Did you enjoy this article or need an experienced Kubernetes Developer for a remote contract-based role? Feel free to contact me.

[WhatsApp](https://www.omkar.cloud/l/whatsapp)

<!--  
Some of my brilliant Creations, that you don’t want to miss out are: 

1.  Bose Framework: An Open Source Framework that is Swiss Army Knife for Selenium. Read the docs at https://www.omkar.cloud/bose/ 

2.  Free Tools: Simplify your daily tasks with the Free Omkar Tools, an all-in-one dashboard that offers a range of free utilities. Whether it's formatting JSON, comparing text, or testing RegExp, these tools have got you covered. Say goodbye to unreliable websites and use it at https://www.omkar.cloud/tools/.

3.  Omkar Cloud: My Samurai Lead Generation Software allows you to access over 200M+ LinkedIn leads, enabling you to win clients. It's like Sales Navigator with superpowers. You can use it at https://www.omkar.cloud/.

4.  Omkar Cloud Datasets: Enhance your analysis with meticulously scraped datasets from Stack Overflow, G2, and many more reputable sources. These fresh datasets are readily available for your data analysis at https://www.omkar.cloud/datasets/.

5.  Kubernetes Consultation: Take your DevOps practices to the next level with our team of Ninja Developers. Through our Kubernetes Consultation service, we can help you turbocharge your DevOps processes and enable you to move at the speed of light. Whatsapp us for a consultation at .

-->

Dhanyawad 🙏! Vande Mataram!

How To Get Land Web Development Clients! 💰💰💰

Chetan — Sat, 22 Apr 2023 17:27:21 +0000

🎯 Overview

You are here as you want to find clients for your web development agency.

We have collected some of the best performing strategies to help you land clients for your web development firm.

Let's not beat around the bush, and let me to share with you two great strategies to get clients for your web development agency.

Suggest Interesting Ideas for their Business

One of the surest to way get a conversation started with a prospect is to get interested in them and talk concretely in terms specifically how you could help them grow.

Here's an example of how can be done

Target the People most likely to need Web Development Services, we recommend targeting decision makers with verified emails at companies with size range of 11 to 100 in industries like Consumer Goods and Consumer Electronics as these people will be in more need of Web Development Service.

You can easily find people who fit this criteria using omkar cloud.

Now you should visit each prospect's exisiting website and send a personalized email thats demonstrates effective steps to make their website better. Following is an excellent example that you can use as a basis for your email

Subject: Ideas for {{company_name}}

Hi {{first_name}},
Congrats on having a huge following on Facebook. However when I checked your website [Website URL] it does not match the quality of work you guys are doing.

I identified a few areas where quick wins could be achieved:

1. Adding a feature for customers to get notified when products are back in stock.
2. Adding a blog to enhance your SEO efforts.
3. Implementing a contact form for customers to easily reach out with questions.

I have helped [Similar Company] and they saw a significant boost in revenue from it this year.

Are these ideas you've been considering?

Best,
[Your Name]

The thing that will really boost response rates is following up with prospect. So Follow with the prospect say 2 Days later in case they didn’t respond. Following is an excellent email template that you can use


Hey {{first_name}}, I know that you are a busy person. So just wanted to follow up so that this doesn't get lost
Thanks,
[Your Name]

Reaching Out to Prospects Hiring a Web Developer

If a company which has posted a job vacancy for web development role then their are high chances to convert them into a Customer.

While some companies want to hire in-house, others are open to discussing the option of working with a web development agency, especially if you can demonstrate that you are a great choice for them.

Also in case they are adamant about hiring in-house still you can still offer to provide them with a skilled web developer from your agency on a monthly contract basis.

Here's an example of how it can be done:

Target the People who are hiring for Web Development Role, we recommend targeting decision makers with verified emails at companies with size range of 11 to 100 who are hiring for Web Developer, Web Designer, Frontend Developer Role.

You can easily find people who fit this criteria using omkar cloud.

Now send a personalized email. Following is an excellent email template that you can use

Subject: Hiring [Job Title] at [Company Name]?

Hi [Name],
Saw that you’re hiring [Job Title] at [Company Name].

I wanted to see if you’d be interested in our Web Development as a Service offering. You only pay for results you get.

Something you might be interested in?

Best,
[Your Name]

As a best practice send a followup email to increase your response rates. Following is an excellent email template that you can use

Hey {{first_name}}, I know that you are a busy person. So just wanted to follow up so that this doesn't get lost
Thanks,
[Your Name]

Next Steps

That's all there is to getting clients for your web development agencies, these were 2 battle tested strategies you can use and see results for yourself.

Understand that Knowledge learned but not applied does not bear fruit. So head over to omkar.cloud, discover your leads, and start shooting emails :)

Wish you best of luck!

Dhanyawad 🙏

Some of my brilliant Creations, that you don’t want to miss out are:

Bose Framework: An Open Source Framework that is Swiss Army Knife for Selenium. Read the docs at https://www.omkar.cloud/bose/
Free Tools: Simplify your daily tasks with the Free Omkar Tools, an all-in-one dashboard that offers a range of free utilities. Whether it's formatting JSON, comparing text, or testing RegExp, these tools have got you covered. Say goodbye to unreliable websites and use it at https://www.omkar.cloud/tools/.
Omkar Cloud: My Samurai Lead Generation Software allows you to access over 200M+ LinkedIn leads, enabling you to win clients. It's like Sales Navigator with superpowers. You can use it at https://www.omkar.cloud/.
Omkar Cloud Datasets: Enhance your analysis with meticulously scraped datasets from Stack Overflow, G2, and many more reputable sources. These fresh datasets are readily available for your data analysis at https://www.omkar.cloud/datasets/.
Kubernetes Consultation: Take your DevOps practices to the next level with our team of Ninja Developers. Through our Kubernetes Consultation service, we can help you turbocharge your DevOps processes and enable you to move at the speed of light. Whatsapp us for a consultation at https://www.omkar.cloud/l/whatsapp.

How To Get Clients For Web Development Agencies

Chetan — Sat, 22 Apr 2023 13:58:24 +0000

🎯 Overview

You are here because you want to find clients for your web development agency.

We have collected some of the best performing strategies to help you land clients for your web development firm.

Let's not beat around the bush, and let me to share with you two great strategies to get clients for your web development agency.

Suggest Interesting Ideas for their Business

One of the surest to way get a conversation started with a prospect is to get interested in them and talk concretely in terms specifically how you could help them grow.

Here's an example of how can be done

Target the People most likely to need Web Development Services, we recommend targeting decision makers with verified emails at companies with size range of 11 to 100 in industries like Consumer Goods and Consumer Electronics as these people will be in more need of Web Development Service.

You can easily find people who fit this criteria using omkar cloud.

Now you should visit each prospect's exisiting website and send a personalized email thats demonstrates effective steps to make their website better. Following is an excellent example that you can use as a basis for your email

Subject: Ideas for {{company_name}}

Hi {{first_name}},
Congrats on having a huge following on Facebook. However when I checked your website [Website URL] it does not match the quality of work you guys are doing.

I identified a few areas where quick wins could be achieved:

1. Adding a feature for customers to get notified when products are back in stock.
2. Adding a blog to enhance your SEO efforts.
3. Implementing a contact form for customers to easily reach out with questions.

I have helped [Similar Company] and they saw a significant boost in revenue from it this year.

Are these ideas you've been considering?

Best,
[Your Name]

The thing that will really boost response rates is following up with prospect. So Follow with the prospect say 2 Days later in case they didn’t respond. Following is an excellent email template that you can use


Hey {{first_name}}, I know that you are a busy person. So just wanted to follow up so that this doesn't get lost
Thanks,
[Your Name]

Reaching Out to Prospects Hiring a Web Developer

If a company which has posted a job vacancy for web development role then their are high chances to convert them into a Customer.

While some companies want to hire in-house, others are open to discussing the option of working with a web development agency, especially if you can demonstrate that you are a great choice for them.

Also in case they are adamant about hiring in-house still you can still offer to provide them with a skilled web developer from your agency on a monthly contract basis.

Here's an example of how it can be done:

Target the People who are hiring for Web Development Role, we recommend targeting decision makers with verified emails at companies with size range of 11 to 100 who are hiring for Web Developer, Web Designer, Frontend Developer Role.

You can easily find people who fit this criteria using omkar cloud.

Now send a personalized email. Following is an excellent email template that you can use

Subject: Hiring [Job Title] at [Company Name]?

Hi [Name],
Saw that you’re hiring [Job Title] at [Company Name].

I wanted to see if you’d be interested in our Web Development as a Service offering. You only pay for results you get.

Something you might be interested in?

Best,
[Your Name]

As a best practice send a followup email to increase your response rates. Following is an excellent email template that you can use

Hey {{first_name}}, I know that you are a busy person. So just wanted to follow up so that this doesn't get lost
Thanks,
[Your Name]

Next Steps

That's all there is to getting clients for your web development agencies, these were 2 battle tested strategies you can use and see results for yourself.

Understand that Knowledge learned but not applied does not bear fruit. So head over to omkar.cloud, discover your leads, and start shooting emails :)

Wish you best of luck!

Dhanyawad 🙏

DEV Community: Chetan

🚀 Introducing ✨ Bose Framework - The Swiss Army Knife for Bot Developers 🤖

Getting Started

Core features

Usage

Configuration

Exception handling

Debugging

task_info.json

final.png

page.html

error.log

Outputting Data

Undetected Driver

LocalStorage

Boss Driver

Learn More

If Bose Framework helped in Bot Development, please take a moment to star the repository. Your act of starring will help developers in discovering our Repository and contribute towards helping fellow developers in Bot Development. Dhanyawad 🙏! Vande Mataram!

Youtube Collabration?

🌈Omkar DevTools: A Swiss Army knife for Ninja Developers🚀

Introduction

How to use DevTools

Final Thoughts

🌈 A One Person Startup Tech Stack for Ninja Developers - Next.js, Django, Kubernetes, and GCP 🚀

Introduction

My Application Omkar Cloud

Tech Stack of Omkar Cloud

Frontend**

Backend

Blog

Deployment

Other Tools

Final Thoughts

🔥💥🚀 10 Missing Selenium Methods for Ninja Developers!😱💥🔥

Introduction

Methods

How to use these utilities?

🚀 🤖💻🔍 How to scrape g2 using Python, Selenium and Bose Framework 🅶2️⃣🐍🖥️

Introduction

Installation

Usage

🤔🌐🗺️ How to scrape google maps using Python, Selenium and Bose Framework 🐍🔍🌍

Introduction

How to scrape google maps

Installation

⭐🤖🔑 9 Ninja Tips to prevent Bot from getting Detected like Grey Ninja 🕵️‍♂️🔍🛡️

Introduction

Tips

1. User-Agent and Window Size Rotation

2. Change your IPs

Free Methods

Paid Method

3. Humane Page Flow

4. Use Random Waits and Slow the Bot

5. Use Captcha Solving Services

6. Use Ultrafunkamsterdam’s ChromeDriver

7. Do not use a Headless Browser

8. Maintain Consistency in User Agent, Window Size, and IP Address

9. Use Bose Framework

💻🎯 Ninja Kubernetes Cheat Sheet for Ninja Developers 📜

A small request

Popular

Pods

Nodes

Services

Deployments

ReplicaSets

StatefulSets

Jobs

CronJobs

ConfigMaps

Secrets

Ingresses

PersistentVolumes

PersistentVolumesClaims

🌍🚀🎯 From Localhost to Cloud ☁️: Next.js, Django, SSL 🔒, GitHub Actions 🚀, DNS| Ultimate Website Deployment Tutorial🌟🔥✨

A small request

Introduction

What we will do

Prerequisites

`task_info.json`

`final.png`

`page.html`

`error.log`