DEV Community: Ella (she/her/elle)

How we moved Crawlee for Python out of Beta

Ella (she/her/elle) — Tue, 30 Sep 2025 08:59:42 +0000

Today is the day we move Crawlee for Python out of beta 🥳 In this post I will summarize the decisions and development that went into this process. I hope it will inform — and even validate — your product decisions.

What is Crawlee for Python?

To begin with, a quick intro: Crawlee for Python is an open-source web-scraping library that provides you with a full toolkit to build, run, and scale your own scrapers. You can deploy Crawlee for Python to extract financial data to identify cases of corporate wage theft, or to track major global polluters, or compare the prices of competing e-commerce retailers. The only real limit is your creativity.

Why was it in beta?

Crawlee for Python was released in beta in July 2024. A couple of years earlier, we had released Crawlee for JS, which had better capabilities from the outset as a result of the tooling that was available for us to build with.

We knew we ultimately wanted both versions of Crawlee to have equivalent capabilities, but the Python tooling available to us at the time was too young and untested in real scenarios. We believed we could reach the point where Crawlee for Python was equivalent to (or better than!) its JS counterpart but weren’t ready to include untested tooling in the build. After working hard to launch, we made the decision to release Crawlee for Python in beta.

This decision gave us breathing room to collect real-world feedback to solve the issues related to any new code and gather community feedback. Getting the product out in beta allowed devs to engage with it, and gave us the chance to fix reported bugs, implement requested features, and improve some performance issues. This helped identify new issues with the library that we could incorporate into a more complete build

What did we do to prepare for v1?

We used the time Crawlee for Python was in beta to improve more than just the library. We created several benchmarks to track the performance of a wide range of typical crawlers, allowing us to compare Crawlee for Python with other options available to developers, including Crawlee for JS.

We re-implemented most of the capabilities of Crawlee for JS in our v1 of Crawlee for Python. Key features we added are:

Unified storage client system, resulting in less duplication, better extensibility, and a cleaner developer experience. It also opens the door for the community to build and share their own storage client implementations tailored to specific databases or cloud storage providers.
Adaptive Playwright crawler, which makes your crawls faster and cheaper, while still allowing you to reliably handle complex, dynamic websites. In practice, you get the best of both worlds: speed on simple pages and robustness on modern, JavaScript-heavy sites.
New default HTTP client (ImpitHttpClient, powered by the Impit library). This delivers fewer false positives, more resilient crawls, and less need for complicated workarounds. Impit is also developed as an open-source project by Apify, so you can dive into the internals or contribute improvements yourself: you can also create your own instance, configure it to your needs (e.g. enable HTTP/3 or choose a specific browser profile), and pass it into your crawler.
Sitemap request loader, making it easier to start large-scale crawls where sitemaps already provide full coverage of the site
Robots exclusion standard which not only helps you build ethical and responsible crawlers, but can also save time and bandwidth by skipping disallowed or irrelevant pages
Fingerprinting so that each crawler run looks like a real browser on a real device, making it less likely that you’ll get blocked. Using fingerprinting in Crawlee is straightforward: create a fingerprint generator with your desired options and pass it to the crawler.
Open telemetry, allowing you to monitor real-time dashboards or analyze traces to understand crawler performance. easier to integrate Crawlee into existing monitoring pipelines

How did we know it was ready to move out of beta?

Once we implemented the features above, and tested them for as many possible real-world cases we believed are possible, we eventually had a version of Crawlee for Python that lived up to our initial vision for it. We feel now is the time: Crawlee for Python is ready to sit alongside its JS counterpart as a full web-scraping and automation library.

Are you ready to check it out? Visit our repo and give us a star ⭐

What happens next?

We will continue to build upon Crawlee for Python v1 as a rolling release. Although we are happy to be launching the full version, we know we can make further improvements. This is where you come in!

We’d love your feedback. Try it out, and don’t hesitate to open an issue to:

Report any bugs or weird behaviour you encounter;
Ask questions if something isn’t clear - so we can improve the docs;
Or request new features you’d like to see

Thanks for reading!

We hope this summary gave you some insights into the process and decisions that take place to move a product out of beta. We would love to hear from you if you have questions about Crawlee for Python (or JS!), the tooling we have implemented, or anything else we have discussed in this article.

Give it a run and share your issues with us so we can continue to refine this tool and adapt it to suit all your web-crawling and automation needs.

How to write a high quality post on DEV

Ella (she/her/elle) — Fri, 20 Aug 2021 13:56:54 +0000

Photo by Daniel Thomas

Writing your first post on DEV can be as unfussy as clicking on the Create Post button at the top of the screen (which takes you to https://dev.to/new) and jotting down some thoughts before hitting the juicy Publish button at the bottom of the screen.

That will get you started and on your way to publishing regularly, which is the key to creating connections with readers who engage with your work.

If you want to build your readership and engagement further, here are a few more tips and tricks to posting on DEV that you might want to know about:

How the editor works

This heading is a little misleading, because there are 2 editors to choose from right now. They're both markdown-based, but look and function a little different from each other:

There's Rich + Markdown

And Basic

You can switch between these in your User Settings under Customization (where you can also try out the different colour schemes featured in those screenshots - but I digress...), just bear in mind that posts previously published in one editor will continue to use that editor for any edits in the future.

While the Basic editor carries the post's meta-data in its header, the Rich version tucks it in around the edges. So, if you want a cover image in the Basic editor, you'll have to link to it in the header. Similarly, if you want to create a series or link to a canonical URL you'll need to add those in yourself.

We recommend reviewing Habdul's excellent, comprehensive guide to posting on DEV. Habdul builds on the Editor Guide in enough detail that you'll be a competent markdown writer and editor by the end of the post.

How to tag (and why you should)

Did you know that top posts under certain tags are eligible for badges each week? Additionally, having your post tagged correctly helps your audience find you, and avoids getting your post reported for suspicious behaviour.

Be sure to check out the tag descriptions for any tags you're planning to use and be sure you're following any posting guidelines. See #mentalhealth or #opensource for examples of tags that have descriptions and guidelines.

Considering accessibility

For clarity on what we mean when we talk about accessibility (or a11y for keyboard-short), Eevis's post on a11y is a good starting point for some general concepts, such as inclusive language. If you'd like to know a little more about why a11y is important, Up Your A11y has a great intro that we'd recommend you check out.

For the endlessly-curious, Ben explains a bit more about why some ways of sharing code in your post are more accessible than others, while Alfie shares some best practices that might help with writing more accessible posts, and Nero starts an important conversation about how accessible emojis are.

Let's talk about images

A picture speaks a thousand words, or something, right? Well, that depends on your alt text...

Not only do pictures help break up the wall-of-text effect of an in-depth tutorial, but they can also help influence the energy of your post and impact on your audience. Just remember to include alt text, so that anyone unable to view the image still knows what it is. For example, the alt text for this image is An audience in an atmospheric auditorium:

Photo by Luis Quintero

While a fix is on the way, cover images currently don't have alt text, so while they're handy for positioning your post at or near the top of the feed, don't rely on the cover image itself to convey any information that's important to the message of your post. That means, even if you have a pretty graphic title for a cover image you still need to write that title out in full.

Speaking of titles

We at DEV always recommend taking some extra time to choose a title that sets your post apart from the rest. There are only so many "The best 10 JavaScript tools" posts any dev has time for, so what about "How this one JS tool changed my workflow and freed up time to grow a herb garden"? The key here is to actually deliver on your title, rather than use it as click-bait to draw in views. Photos of your herb garden (with appropriate alt text) help:

And finally

We are here to learn from each other. If you have any other hints, tips, or best practices for post authors please feel free to share them below! We'd love to learn about the things you've discovered here on DEV and how you connect with your readers.

Remember that all posts should abide by the Terms of Use and Code of Conduct. This includes inclusive language, original work that you have the right to share (with credit to anyone else whose original work you're citing or referencing), and not posting for the purpose of producing backlinks or driving traffic to an external site.

Hello DEV community! I just joined the Forem team

Ella (she/her/elle) — Fri, 12 Feb 2021 17:24:22 +0000

Hiya DEV peeps, I'm Ella and I'm a dev.to lurker. As a tech-positive non-developer, I've been enjoying the richness of this community and how supportive and collaborative y'all are. It's such a rush to write my first post here, announcing that I've joined the fantastic team behind DEV and Forem as a Community Associate.

What does this mean?

I am beyond excited to be joining @michaeltharrington and @itscasey to support everyone on dev.to, community.codenewbie.org, and the Forem ecosystem. We're here to help, so don't hesitate to reach out to us at yo@dev.to any time!

Despite my first forays into coding 20+ years ago, curiosity took me down a different path so my career background is in face-to-face communities, wellness, and the ubiquitous service industry. My goal is to bring empathy and compassion from these fields into my role here, and serve this community and all the Forems with heart, soul, and a healthy dose of humility.

Why Forem?

As a former youth worker and community educator, I have a deep stake in a healthier internet. DEV is a model for inclusive online communities that encourage and build people up, share knowledge and insights that narrow the education gap, and elevate everyone's access to learning and growth opportunities. It's incredible that the wider world now gets a chance to build online spaces in its image.

I gotta hand it to you all, because the community here is what makes all of this possible. Thank you, DEV members, for contributing to making this place great!

By the by

I'm into dancing, callisthenics, growing plants (indoors and outdoors), knitting, and playing too much League of Legends. If any of this resonates, feel free to connect!