DEV Community: rajesh singh

Vote for Name of a Bot Development Framework for Python.

rajesh singh — Tue, 11 Jul 2023 09:49:33 +0000

Hi,
I want to know which name you think is a Good Name for a Bot Development Framework. The Options are

Scrapium
Botasaurus
Bose Framework

Please Vote for your favorite name here

Which frontend & backend framework are you using for your application?

rajesh singh — Wed, 14 Jun 2023 03:59:29 +0000

Hello there!

Which frontend & backend framework are you using for your application?

Meet Omkar DevTools: 🚀 Your Swiss Army Knife as a Ninja Developer ✨

rajesh singh — Mon, 12 Jun 2023 16:08:54 +0000

💡 What is Omkar DevTools?

Omkar DevTools is a one-stop shop for developers, providing a comprehensive set of tools to help with tasks from Lorem Generation to JSON Decoding under a single dashboard.

✨ Top Tools

Base 64 Encoder
JWT Decoder
JSON Duplicate Remover
Hash Generator
Lorem Generator
Secret Generator (Secure for use as secrets in Django, JWT keys)
Text Counter
Case Converter

And many more tools to save your valuable development time. 👨🏻‍💻

🚀 Get Started

Get Started with Omkar DevTools by visiting this link.

Also, bookmark Omkar DevTools for easy single click access when you need it.

👋 Hi Reader,

What do you think? Do you see the value of Omkar DevTools?

Share your thoughts in the comments and I will reply to every single comment.

Meet Bose Framework: 🚀 Your Swiss Army Knife as a Ninja Scraper ✨

rajesh singh — Mon, 12 Jun 2023 15:59:06 +0000

💡 What is Bose Framework?

Bose Framework is a framework built for selenium developers packed with best practices from experienced bot developers to help create undetectable, low boilerplate and easy to debug bots. 🤖

It helps you scrape or automate target websites with ease and gives you the mental powers of experienced bot developers at your fingertips, saving you hours of development time. 👨🏻‍💻

🏆 Top Feautres

Bose is a battery packed framework that implements following feautres to help you create bots faster, saving your valuable Development Time. ✨

Go Low Boilerplate with Bose Launching a browser using Bare Selenium requires writing a significant amount of code:

from selenium import webdriver

driver_path = 'path/to/chromedriver'

driver = webdriver.Chrome(executable_path=driver_path)

driver.get('https://www.example.com')

driver.quit()

With Bose, you can launch browser in few lines of code without having to worry about specifying paths:

from bose import *

class Task(BaseTask):
    def run(self, driver):
        driver.get('https://www.example.com')

Configure Profile, Window Size, and User Agent with a Single Line of Code

In bare Selenium, if you want to configure options such as the profile, user agent, or window size, it requires writing a lot of code, as shown below:

from selenium.webdriver.chrome.options import Options
from selenium import webdriver

driver_path = 'path/to/chromedriver.exe'

options = Options()

profile_path = '1'

options.add_argument(f'--user-data-dir={profile_path}')

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37")'
options.add_argument(f'--user-agent={user_agent}')

window_width = 1200
window_height = 720
options.add_argument(f'--window-size={window_width},{window_height}')

driver = webdriver.Chrome(executable_path=driver_path, options=options)

With Bose, you can specify them in a single line of code using predefined variables. Here's an example:


class Task(BaseTask):
    browser_config = BrowserConfig(user_agent=UserAgent.user_agent_106, window_size=WindowSize.window_size_1280_720, profile=1)

Exception Handling that every Bot Developer wants

Bose also addresses a common frustration with Selenium - when an exception occurs, the browser crashes and closes automatically. This behavior is not desirable for bot developers and makes debugging hard.

Bose, on the other hand, keeps the browser open in the event of an exception, allowing for easier debugging of the problem.

Remembers the Past Runs

Let's say you went to drink coffee while your bot was running, and when you came back, you noticed that the bot had closed.

As a developer, you might have an itch to see the last screenshot taken before the browser was closed or to know how much time the bot took to complete its task.

Well, Whenever you run Bose, it automatically stores important details such as

the screenshot taken before closing
the time it took to run the bot
any errors that occurred.

After, each run a new folder is created in tasks folder which contains three files, listed below:

`task_info.json`

It contains information about the task run such as

duration for which the task run,
the ip details of task
the user agent
window_size
profile

`final.png`

This is the screenshot captured before driver was closed.

`page.html`

This is the html source captured before driver was closed. Very useful to know in case your selectors failed to select elements.

`error.log`

In case your task crashed due to exception we also store error.log which contains the error due to which the task crashed. This is very helful in debugging.

Built to defeat Cloudflare, Perimeterx by default

Multi Million Dollar Companies protects their valuable data with the help of Cloudflare and PerimeterX.

Now, you might be wondering how to bypass systems like Cloudflare and PerimeterX. Well, a brilliant Developer named Leon created a ChromeDriver that has excellent support for bypassing all major bot detection systems such as Distil, Datadome, Cloudflare, and others.

To use it, simply pass the use_undetected_driver option to the BrowserConfig in your code, as shown below:

from bose import BaseTask, BrowserConfig

class Task(BaseTask):
    browser_config = BrowserConfig(use_undetected_driver=True)

Output Data in CSV or JSON with a Single Line of Code

Outputting data in CSV or JSON requires a significant amount of imperative code, as shown below:

import csv
import json

def write_json(data, filename):
    with open(filename, 'w') as fp:
        json.dump(data, fp, indent=4)

def write_csv(data, filename):
    with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
        fieldnames = data[0].keys()  # get the fieldnames from the first dictionary
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writeheader()  # write the header row
        writer.writerows(data)  # write each row of data

data = [
    {
        "text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d",
        "author": "Albert Einstein"
    },
    {
        "text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d",
        "author": "J.K. Rowling"
    }
]

write_json(data, "data.json")
write_csv(data, "data.csv")

Bose simplifies these complexities by encapsulating them in the Output module for easy reading and writing of data:

from bose import Output

data = [
    {
        "text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d",
        "author": "Albert Einstein"
    },
    {
        "text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d",
        "author": "J.K. Rowling"
    }
]

Output.write_json(data, "data.json")
Output.write_csv(data, "data.csv")

Run the Same Code Everywhere, whether it's on Mac, Linux, or Windows. Forget the need to change driver paths.

Bose simplifies cross-platform development by abstracting away the differences between operating systems such as Windows, Mac, and Linux.

You no longer need to specify driver paths specific to each OS when launching browser.

Adds Powerful Methods to Supercharge Bot Development

The driver you receive in the run method of the Bose Task is an extended version of Selenium that adds powerful methods to make working with Selenium much easier. Some of the popular methods added to the Selenium driver by Bose Framework are:

METHOD	DESCRIPTION
get_by_current_page_referrer(link, wait=None)	simulate a visit that appears as if you arrived at the page by clicking a link. This approach creates a more natural and less detectable browsing behavior.
js_click(element)	enables you to click on an element using JavaScript, bypassing any interceptions(ElementClickInterceptedException) from pop-ups or alerts
get_cookies_and_local_storage_dict()	returns a dictionary containing "cookies" and "local_storage”
add_cookies_and_local_storage_dict(self, site_data)	adds both cookies and local storage data to the current web site
organic_get(link, wait=None)	visits google and then visits the “link” making it less detectable
local_storage	returns an instance of the LocalStorage module for interacting with the browser's local storage in an easy to use manner
save_screenshot(filename=None)	save a screenshot of the current web page to a file in tasks/ directory
short_random_sleep() and long_random_sleep():	sleep for a random amount of time, either between 2 and 4 seconds (short) or between 6 and 9 seconds (long)
get_element_or_* [eg: get_element_or_none, get_element_or_none_by_selector, get_element_by_id, get_element_or_none_by_text_contains,]	find web elements on the page based on different criteria. They return the web element if it exists, or None if it doesn't.
is_in_page(target, wait=None, raise_exception=False)	checks if the browser is in the specified page

In simple words, Bose is an excellent framework that simplifies the boring parts of Selenium and web scraping.

🚀 Get Started with Bose

Now, let's see how you can have the magic of Bose at your finger tips.

Start by Cloning the Template

git clone https://github.com/omkarcloud/bose-starter my-bose-project

Then change into that directory, install dependencies, open vscode, and start the project:

cd my-bose-project
python -m pip install -r requirements.txt
code .
python main.py

The first run will take some time as it downloads the chrome driver executable, subsequent runs will be fast.

Once started it will scrape data from quotes.toscrape.com and store the results in /output/finished.json

✨ Upcoming Features in V2 to supercharge 🔋 Bot Development

Kubernetes Integration to help you scrape data at Google’s Scale [Priority]
Save Storage by storing the profile in a single JSON file by storing cookies and local storage for the website. [Priority]
Provide a temporary email service [Priority]
Purchase hundreds of pre-created Google and Microsoft accounts [Priority]
Built-in IP rotation for requests [Priority]
Captcha Solving implemented right into Bose [Priority]
Generate names, emails, usernames, etc., for users in countries such as India, Russia, Europe, China, and America.

📚 Summary

Simply put, Bose empowers you to effortlessly automate or scrape your Target Website and its content with the ease of cutting butter with a knife.

👋 Hi Reader,

What do you think? Do you see the value of Bose Framework?

Share your thoughts in the comments and I will reply to every single comment.

Meet Om Startup Framework: 🚀 Your Swiss Army Knife as a Full Stack Ninja ✨

rajesh singh — Mon, 12 Jun 2023 15:56:53 +0000

💡 What is Om Startup Framework?

The Om Startup Framework helps you create production-ready web applications using Next.js and Django.

It's a battery-packed framework that implements the following features to help you take your idea from concept to launch in a matter of days, saving developers months of development time.

Production Ready Blogging System

Production Ready Authentication Pages with Google Auth, Email Auth, Email Verifivation and Forgot Password Functionality.

Responsive and Beautiful Pricing Page with Whatsapp and FAQ Button which also supports Variable Pricing for Different Countries

User Management Dashboard with ability to Delete or Ban users of your Application

Stunning Landing Page

User Action Tracking so you know exactly what each user did in your application from sign in to sign out and everything in between.
Ready Made Url Shortner to create short urls for tracking clicks. Say Sayonara to bit.ly.

Pre configured Github Actions so delpoying new code is as simple as a git push
Contact Us Functionality which also sends a Thank You Email for Contacting Us.
Brillant Privacy Policy and Terms of Conditions to safeguard your StartUp.
Great Guidance of Deployment and Important Tools you need for your StartUp.

🚀 Get Started

Installation

Clone Starter Template

git clone https://github.com/omkarcloud/om-startup-framework-starter my-om-project
cd my-om-project

Then change into frontend/ directory and install dependencies.

cd frontend/
npm install --legacy-peer-deps

Then change into blog/ directory and install dependencies.

cd blog/
npm install

Then change into backend/ directory, install dependencies, migrate database and seed with data.

cd backend/
python -m pip install -r requirements.txt
npm run db:delete-and-seed

Run the Frontend and Backend Server

Start the frontend server

cd frontend/
npm run dev

In a seperate terminal start the backend server

cd backend/
npm run dev

Now, visit http://localhost:3000 to see the home page with a nav bar linking to various pages of application. Explore the pages by clicking on navigation links to get a big picture of feautres of your application.

Run the Blog Server

Most of your time will be spent writing frontend and backend code. Occasionally, you will need to run the blog server to preview the articles you write. Run following Command to spin up the blog server

cd blog/
npm run dev

Now visit http://localhost:4000/blog/ to see the blog home page.

👨🏻‍💻 Pre-Requisite Knowledge

To get the most out of our docs, it's recommended that you have a basic understanding of Next.js and Django.

Given that you know React, If you are interested to learn Next.js and Django, these resources are Great Starting Point:

Next.js: Next.js Documentation [Free]
Django: Mosh Hamedani Django Course [Paid]

✨ Upcoming Features in V2

Support for affiliate marketing [Priority]
Port framework to SvelteKit, Nuxt.js, and Nest.js to increase reach [Priority]
Analytics dashboard like Google Analytics
Chat support

📚 Summary

Simply put, Om Startup Framework holds your hand, from authentication to deploying your app to the world

If you want to save yourself months of development time, I highly encourage you to invest your time in learning about the Om Startup Framework by visiting this link.

👋 Hi Reader,

What do you think? Do you see the value of Om Startup Framework?

Share your thoughts in the comments and I will reply to every single comment.