Arindam Majumder

Posted on Dec 12

7 Must-Try Open-Source Tools for Python and JavaScript Developers 🚀

#javascript #webdev #python #beginners

Open-source tools are something almost every developer uses, whether while learning to code or building real-world apps.

Python recently became the most popular language on GitHub surpassing JavaScript. Now both are the top two programming languages in open source space.

In this article, I’ll share 7 open-source tools for Python and JavaScript developers that can help you build better projects.

Let's dive in!🚀

You can use these tools in your projects or even contribute to them to make them better.

Crawlee - Build reliable crawlers

Crawlee is an open-source library for web scraping and browser automation. It supports HTTP requests and headless browser crawling with tools like Puppeteer and Playwright.

Its anti-blocking features, such as proxy rotation and browser fingerprinting, make it easier to avoid detection. Crawlee also scales automatically and supports TypeScript, making it reliable and efficient for tasks of any size.

The best part? Crawlee works seamlessly with both Node.js and Python. If a website requires JavaScript rendering, you can easily switch to a browser-based crawler. Later, if you discover a faster API for your needs, simply switch back to HTTP crawling without rewriting your project

To run Crawlee quick start, install Crawlee CLI with the necessary dependencies and boilerplate codes:

npx crawlee create my-crawler

Crawlee comes with three main crawler classes: CheerioCrawler, PuppeteerCrawler and PlaywrightCrawler having same interface.

You can run Crawlee in your project by using:

npm install crawlee

Let’s take a look at example to “Crawl all links on a website” using Crawlee:

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, enqueueLinks, log }) {
        log.info(request.url);
        // Add all links from page to RequestQueue
        await enqueueLinks();
    },
    maxRequestsPerCrawl: 10, // Limitation for only 10 requests (do not use if you want to crawl all links)
});

// Run the crawler with initial request
await crawler.run(['https://crawlee.dev']);

This example uses the enqueueLinks() method to add new links to the RequestQueue as the crawler moves through pages. By removing the maxRequestsPerCrawl option, you can use it to find all URLs on a domain.

You can find more examples and resources to properly use Crawlee inside your project from its documentation.

They have 15k+ stars on GitHub, project is constantly evolving and improving.

Star Crawlee on GitHub ⭐️

🚨 Giveaway Alert! 🚨

Crawlee is giving away 3 exclusive hoodies to the community.

Click the button below to Star Crawlee on GitHub.
Drop your GitHub username in the comments section of this article.

That’s it!

Crawlee will select the lucky winners on December 31st, 2024.

Particiapate in the Giveaway

Marimo - Reactive notebook for Python

Marimo is an open source Python-based notebook environment that makes working with data more interactive and intuitive. It supports both Python scripts and notebooks, allowing for seamless version control through Git.

Marimo’s standout feature is its reactive UI, where changes in one part of the notebook automatically update other dependent parts, making it ideal for real-time data exploration.

It is easy to use with plenty of features for advanced users. For example, it can be used to create interactive tools like embedding visualizers.

You can Quickstart Marimo by running this Installation command in terminal:

pip install marimo  # or conda install -c conda-forge marimo
marimo tutorial intro

Create or edit notebooks with:

marimo edit

Run apps:

marimo run your_notebook.py

Best part, with Marimo:

Run Your Notebook as a Web App: Transform your notebook into an interactive web app, by keeping the Python code hidden.
Execute as Scripts: Run your notebook as a script directly from the command line.
Automatic Conversion: Convert Jupyter notebooks to marimo notebooks automatically using the CLI.

You can learn more about Marimo and find resources to properly use it in your next Python workflow through its documentation.

They have 8k+ stars on GitHub, and community is growing fast!

Star Marimo on GitHub ⭐️

Deno - JavaScript runtime for the modern web

Deno is an open-source JavaScript runtime, designed to provide a modern and secure developer experience. Built on V8 and Rust, Deno is optimized for simplicity and security by default.

Install Deno on your Mac or Linux by running single command:

curl -fsSL https://deno.land/install.sh | sh

For Windows, run this:

irm https://deno.land/install.ps1 | iex

It also supports TypeScript natively and includes built-in tools for tasks like code linting, formatting, testing, and compiling into standalone executables.

Even though Deno is quite different from Node.js, it offers a Node.js compatibility layer, allowing you to run modern Node.js projects with minimal changes. You can also migrate from Node.js to Deno, making your code more simple and efficient.

Deno enforces a strict permission model, limiting access to files, networks, and environments unless explicitly granted. Its comprehensive APIs and minimal configuration make it a powerful choice for cloud and edge deployments, offering a scalable environment for building and running applications.

You can learn more about Deno and find resources to properly use it in your workflow through its documentation.

They have 98k+ stars on GitHub, and thousands of developers are using Deno!

dStar Deno on GitHub ⭐️

Wasp - Faster full-stack web apps

Wasp is an open source framework designed to streamline full-stack web development. You can easily built on top of React, Node.js, and Prisma via Wasp.

It simplifies the setup and orchestration of modern web apps by handling boilerplate code and configuration for you. With its simple config language, Wasp allows developers to focus on their app's core logic while automating repetitive tasks, making development faster and more efficient.

For Quick start Wasp, run this installation command on Linux / OSX / WSL (Windows):

curl -sSL https://get.wasp-lang.dev/installer.sh | sh

Create new app:

wasp new

Now, run the Wasp app:

cd <my-project-name>
wasp start

You can easily see the full-stack web app at http://localhost:3000/

If you want to see how easily you can use Wasp for your next full-stack project, check out their tutorial here.

You can learn more about Wasp framework and find resources to properly use it in your workflow through its documentation.

They have 13k+ stars on GitHub, and have a good community support as well!

Star Wasp on GitHub ⭐️

PydanticAI - Agent framework

PydanticAI is a new agent framework designed to simplify Generative AI integrations into your production grade Python applications.

Some standout features:

Structured Output Validation: It ensures AI-generated outputs match predefined data schemas, minimizing unexpected results.
Dependency Management: Provides tools to easily manage and use dependencies, whether synchronous or asynchronous.
Model-agnostic: Supports various AI models like OpenAI's, Groq and Gemini, allowing seamless adaptation for different use cases

Building on the strengths of Pydantic and the validation layer of the OpenAI SDK, the Anthropic SDK, LangChain, LlamaIndex, AutoGPT, Transformers, CrewAI, Instructor and many more.

PydanticAI is available on PyPI, so you can install by running:

pip install pydantic-ai

You can use it with optional Logfire integration or just install examples:

pip install 'pydantic-ai[examples]'

These examples require you to set up authentication with LLMs and follow additional steps, find everything in this Doc.

PydanticAI is a new agent framework, and some features are still in beta. If you’re planning to try it, you can get involved at an early stage.

You can learn more about PydanticAI agent framework and find resources to properly use it in your workflow through its documentation.

They have 1.3k+ stars on GitHub, and great backing of strong Pydantic community!

Star PydanticAI on GitHub ⭐️

PocketBase - Backend in 1 file

PocketBase is an open source self-hosted backend solution designed for building full-stack applications with a focus on simplicity and portability.

It provides essential features like authentication, file storage, and real-time CRUD operations, all within a single server instance. Ideal for small to mid-sized applications such as mobile app backends and SaaS platforms.

Some great feature includes:

Support of multiple frontend stack like Svelte, Angular, react and others
Javascript and Dart SDK
Self Hosted

PocketBase was created to assist building self-contained applications. It's a great choice for projects that need a lightweight backend without the complexities of managing multiple services.

This project is completely open-source and maintained by volunteers without any funding. You can get involved or explore its features and implementation details from documentation.

They have 41k+ stars on GitHub, and maintained by part time contributors.

Star PocketBase on GitHub ⭐️

KitOps - ML collaboration made easy

KitOps is an open-source tool that redefines how data scientists, developers, and SREs collaborate on AI/ML models, datasets, and source code.

By leveraging Modelkit, a OCI (Open Container Initiative) standards packaging format, KitOps offers a unified approach to packaging, versioning, and deploying resources across diverse environments.

To use KitOps, you need to install Kit CLI and check version by running:

kit version

kit login jozu.ml

After login, pull a sample Modelkit from Jozu Hub and try it inside your workflow by running the following command:

kit unpack jozu.ml/jozu-quickstarts/fine-tuning:latest

The unpack command will unpack the ModelKit contents to the current directory by default and you can play with it to find out more.

Key Features

OCI-Compliant Packaging: Package models, datasets, configurations, and code into ModelKits for streamlined versioning.
Registry Flexibility: Compatible with any OCI-compliant registry, ensuring seamless integration with existing AI/ML workflows.
YAML-Based Customization: Configure pipelines, fine-tune LLMs, and build RAG pipelines with intuitive YAML files.

The best part?

If you're a data scientist working on Python scripts in Jupyter Notebooks with a team of AI engineers building with LLM models, KitOps bridges the gap.

It enables seamless collaboration across teams, combining diverse workflows into a unified pipeline for maximum efficiency.

You can learn more about KitOps and find resources to properly use it in your workflow through its documentation.

They have 500+ stars on GitHub, and the community is growing!

Star KitOps on GitHub ⭐️

That’s it! I've tried to cover some helpful tools mainly focused on Python and JavaScript developers. If you know other awesome tools, write them down in the comments!

Hope you found this article useful. If so, feel free to share it with your developer friends!

Also, Follow me For More Content like this: