DEV Community: Natalia Demianenko

A Comprehensive Guide to Finding the Perfect Web Scraper

Natalia Demianenko — Tue, 27 Aug 2024 12:21:30 +0000

Introduction

Web scraping is a powerful tool that lets you collect data from websites automatically. Whether you’re tracking prices, gathering research data, or monitoring competitors, web scraping can save you time and effort. However, finding the right tool for the job can be tricky. With so many options available, it’s important to choose the one that best fits your needs. In this guide, we’ll help you understand what web scraping is, what types of scrapers exist, and how to find the perfect tool for your projects.

Understanding Web Scraping

Web scraping is the process of automatically extracting data from websites. Instead of manually copying and pasting information, a web scraper does the work for you, collecting the data you need quickly and efficiently. This data can then be used for various purposes, like analyzing market trends, gathering research material, or populating databases.

Types of Web Scrapers

Code-Based Scrapers:
These are tools that require programming knowledge. Developers often use languages like Python with libraries such as BeautifulSoup or Scrapy to build custom scrapers. These tools are highly flexible but require some coding skills.
No-Code Scrapers:
No-code scrapers are user-friendly tools designed for people who don’t know how to code. Tools like Octoparse and ParseHub allow users to extract data through simple point-and-click interfaces, making them accessible to beginners.
Browser Extensions:
These are lightweight tools that can be added directly to your browser. Examples include DataMiner and Web Scraper, which allow you to scrape data directly from the webpage you’re viewing.
API-Based Scrapers:
Some websites offer APIs (Application Programming Interfaces) that let you retrieve data in a structured format without scraping. API-based scrapers use these APIs to collect data more efficiently and reliably.
Custom-Built Scrapers:
These are specialized tools created for a specific task or website. They are tailored to handle unique data extraction needs, offering the most customization but often requiring significant development effort.

How to Find the Perfect Web Scraping Tool

Identifying Your Needs
Before choosing a web scraper, it’s important to clearly define what you need. Ask yourself:

What type of data are you collecting? Determine whether you need to scrape text, images, or structured data like tables.
How much data do you need? Consider whether you need to scrape large amounts of data regularly or just occasionally.
What’s your technical skill level? Decide if you need a no-code tool or if you’re comfortable using a code-based solution.

Researching Available Tools
Once you know your needs, start researching tools that match them. Here are some tips:

Use Online Directories and Reviews: Look for directories or websites that list and review different web scraping tools. Compare features, pricing, and user experiences to narrow down your options.
Try Before You Buy: Many tools offer free trials or demos. Take advantage of these to test how well they meet your requirements before committing.

Evaluating Features
When comparing tools, focus on the features that matter most for your project:

Data Export Options: Check if the tool can export data in the format you need, such as CSV, JSON, or Excel.
Scheduling and Automation: If you need to scrape data regularly, look for tools that offer scheduling and automation features.
Customization: Some tools allow you to customize how data is extracted, filtered, and processed. Make sure the tool you choose offers the flexibility you need. By carefully considering these factors, you can find a web scraping tool that’s perfectly suited to your specific project, ensuring that you get the data you need efficiently and effectively.

Introducing the All Scraping Tools Directory

I'm excited to introduce you to our web scraping solutions directory!🚀
https://allscrapingtools.com/
I hope this aggregator of scraping tools will save others time and make it easier to find what they need 🙏

What is the All Scraping Tools Directory?
The All Scraping Tools Directory is a comprehensive resource designed to help you find the perfect web scraping tool for your needs. It’s an online directory that brings together a wide range of web scrapers, making it easier to compare and choose the right one.

How It Works
The directory is user-friendly and easy to navigate. You can browse through various scraping tools, filter them by categories such as no-code, code-based, or API-based, and see detailed descriptions of each tool’s features. This makes it simple to find a tool that matches your specific requirements, whether you’re a beginner or an experienced developer. We have launched this initial version, but there’s still a lot of work ahead. We plan to add more tools, enhance navigation with additional categories, and expand the content with a blog full of helpful articles and tips. Stay tuned for updates!

Why It’s Useful
Finding the right web scraping tool can be time-consuming, especially with so many options available. The All Scraping Tools Directory saves you time by aggregating all these tools in one place. You can quickly compare features, read user reviews, and make an informed decision without having to search multiple websites. Plus, the directory is regularly updated, ensuring that you have access to the latest tools and features.

By using this directory, you can streamline the process of finding a web scraper, allowing you to focus more on your projects and less on searching for the right tool.

Conclusion

Choosing the right web scraping tool is crucial for successfully gathering the data you need. With so many options available, it’s important to consider your specific requirements, such as the type of data, your technical skills, and the features you need. The All Scraping Tools Directory simplifies this process by providing a comprehensive list of tools in one place, helping you find the perfect solution quickly and easily.

By using the directory, you can save time, reduce the hassle of searching for tools, and ensure that you’re using the best possible scraper for your project. We encourage you to explore the directory, share your feedback, and suggest any tools or features you’d like to see added. Together, we can make this resource even more valuable for the web scraping community.

AI-Enhanced Web Scraping

Natalia Demianenko — Thu, 18 Jul 2024 15:37:24 +0000

Introduction

Web scraping undeniably plays a crucial role in our information society, offering the ability to extract valuable data from various web resources.

In this article, I will share the journey of creating my product — AIScraper, and delve into its key features. By harnessing the power of artificial intelligence, this tool not only automates the process but becomes a true fusion of technologies for rapid and intelligent collection of structured data. But first, let’s dive into the theory and history.

Classical Scraping

Classical scrapers are tools designed to extract data from web pages by analyzing and parsing HTML code. These applications operate based on predefined rules, indicating which elements of the page should be extracted and how they should be interpreted. Despite their popularity, classical scrapers are limited in their ability to adapt to changes in the structure of websites, requiring constant code updates with each alteration on the target page. This need for regular maintenance makes classical scrapers less user-friendly and sometimes inefficient in the dynamic web space.

I have extensive experience creating various scrapers in JavaScript. These tools interacted with both static content, extracting data from HTML using Cheerio or API responses, and dynamic content, requiring page loading using libraries like Puppeteer. From my experience, it became clear how often selectors needed adjustments or even a complete rewrite when the structure of a resource changed.

A similar challenge arises with extensions, where configurations specifying selectors for data collection are necessary. But what can we do with the sites like this?

Each entry is just a paragraph with lines of text.
These lines do not follow the same order, and not every entry contains all the information, so nth-child selectors won't help here.
There are thousands of such entries, making manual work impractical.
There is no API, the data comes in this form directly in the HTML.

Traditional scraper applications and even custom code can't handle this task effectively. But an AIScraper can do it easily! 🪄

Enter AI

With the advent of artificial intelligence, the opportunity to overcome these limitations arises. In my early experiments, I simply handed it HTML and asked it to gather the necessary data in the form of a table — and it worked!

It was at this moment that the idea of creating an extension capable of utilizing these principles for scraping emerged. This is convenient since you are already authenticated on the required resource; there is no need to copy URLs or HTML, take screenshots, and so on. With just a few clicks, you can obtain the desired result without leaving the web page.

Development Kickoff

The first step in development was creating a simple collection in Postman. The initial request involved obtaining the HTML code of the target resource, which was then passed for processing in a subsequent request to the OpenAI API. Having received feedback, I proceeded to create a user-friendly frontend and made adjustments to the backend.

Creating a Demo

As part of the demonstration, I prepared a working extension capable of:

Analyzing a web page to determine extractable data

Finding elements containing the required information

Making on-the-fly modifications (changing the tone of text, correcting grammar, creating summaries)

Collecting data

Thus, the entire scraping process is built on the use of artificial intelligence. You can check out the demo here.

Technical and Financial Aspects

Now my scraper operates on the Anthropic Haiku, it has big context window. But to reduce costs, I preprocess the page before transmitting data, removing unnecessary attributes, comments, scripts, and other technical elements. Selector search is performed on a more limited version of the page (even text is truncated to 20 characters in each element) and is carried out iteratively. This allows passing the page in parts until the required information is found, significantly reducing costs.

Audience Search and Feedback Analysis

However, like any product, it is crucial first to understand what the end user needs and direct the development of functionality based on these needs. I have many ideas, including implementing image recognition, automatic data export to Google Sheets, and much more. I've already made results analysis and created another product - Landing Pages Scraper to collect info about products and services for directory. Also I've recently released API of AIScraper - https://aiscraper.co/api/. I'm still searching for my niche.

I invite you to try using the scraper. I would be grateful for any feedback you can leave directly through the extension’s form or by contacting me on any of the social media platforms.

Conclusion

Creating AIScraper marks a significant step in the evolution of tools for data collection from the internet. Overcoming the limitations of traditional scrapers and extensions, this new tool, harnessing the power of artificial intelligence, demonstrates efficiency in the rapid, intelligent, and automated collection of structured data. But what the final product will be like, we will discover together. You can subscribe to me on Twitter and follow the scraper’s development.

Revolutionary Approach to Information Scraping: Using OpenAI API for Efficient Data extraction

Natalia Demianenko — Thu, 16 Nov 2023 18:10:00 +0000

Introduction

In the world of modern technologies, where data is a key resource, the processes of collecting and analysing data play a crucial role in making informed business decisions. Among the myriad of tools used for this purpose, scrapers stand out, automating the process of scraping information from web resources.

Scrapers are programs that are indispensable tools for the automated collection of data from web pages. They work by analyzing the HTML code of a page and extracting the necessary information, allowing users to obtain up-to-date data.

Who Needs It?

Data Managers
In contemporary companies, where data has become a key asset, data managers play a critical role in the collection, processing, and analysis of information. Their tasks include not only data collection but also ensuring its accuracy, relevance, and structure to support strategic decisions.

Marketers
Marketers can use the product for a deeper analysis of consumer data, as well as for monitoring and analyzing trends on social media and blogs.

Researchers
Researchers can take advantage of the opportunity for more intelligent analysis of large volumes of data when conducting studies in various fields.

Financial Analysts
Financial analysts can use the product to automate the process of analyzing financial data, including news, market reviews, and financial reports.

Limitations of Traditional Scraping Methods

While there are many tools for scraping data, many of them face limitations in directly extracting information from specific elements of web pages. This creates problems when the structure of websites changes and requires constant modification of scrapers to remain effective. Various browser extensions for information collection require a lot of manual intervention, even during initial setup.

In the following sections, we will explore how using OpenAI API can significantly improve the scraping processes, providing more flexible and intelligent capabilities for data managers and other professionals.

Overview of Existing Solutions

I conducted a brief investigation into existing scraping tools and their capabilities. Here is a small list of the most popular Chrome browser extensions that allow configuring data collection from web pages:

These are powerful tools, but they all require extensive initial setup and store selectors of selected elements for data collection, making them prone to reconfiguration if the page layout changes.

Have you used them? Share your experiences in the comments.

Advantages of Using AI in Scraping

Search Across the Entire Page
One of the main advantages of using the OpenAI API in data scraping is the ability to create intelligent algorithms for information retrieval across the entire page. Unlike traditional scrapers, models can consider context and recognize data even if it’s not explicitly highlighted.

Automatic Data Formatting
The OpenAI API provides unique capabilities for automatically formatting extracted data. This can include changing the tone of the text, highlighting key fragments, and even adapting the data structure according to user requirements.

Automated Information Analysis
Imagine a data manager needing to extract information about consumer reactions to a product. With the OpenAI API, the scraper can analyze the entire page content, extract reviews, determine tonality, and highlight key factors influencing the perception of the product.

Adaptability to Changes in Page Structure
Traditional scrapers often lose efficiency when the design or structure of a web resource changes. OpenAI API, trained on diverse data, can adapt to changes, ensuring more stable and productive work.

Just imagine, you don’t have to highlight specific elements on the page from which you need to gather information. You simply list what you want to collect (e.g., “name, rating, review”), and the AI will do it all for you! Of course, some initial setup may be required, such as specifying the page type and the method of navigating through all pages. Additionally, you may need to check and fine-tune the AI scraper if it makes mistakes. However, this is much easier than starting from scratch! You can also ask the AI to change the tone of the collected text, remove mentions of certain words, shorten it to a few sentences, and much more — all done after collecting the information.

I have some developments on applying AI for scraping, and I plan to create such a convenient tool that simplifies and speeds up the information scraping process. Subscribe and follow the progress!

Share your thoughts on this — do you find such a product useful, and would you use it? Have you encountered similar ready-made solutions?

10 Github Repositories for Software Testers

Natalia Demianenko — Sat, 21 Oct 2023 11:28:15 +0000

Software testing is an integral part of the software development process, ensuring that the final product is reliable, bug-free, and meets user expectations. Manual testers play a crucial role in this journey, meticulously evaluating software to identify issues, report defects, and provide valuable feedback. To excel in their roles, testers often rely on a wide range of resources, from testing methodologies and roadmaps to helpful tools and educational materials.

In this article, we've curated a list of 10 Github repositories tailored to the needs of manual testers. These repositories are a treasure trove of information, offering roadmaps for career growth, book recommendations, course listings, and an assortment of tools to simplify the testing process. Whether you're a budding manual tester looking to start your journey or a seasoned professional seeking new resources, this collection will help you navigate the world of software testing and stay on the cutting edge of this dynamic field.

Let's dive into these repositories and explore the wealth of knowledge and tools they offer, helping you become a more effective and informed software tester.

1. Awesome Quality Assurance Roadmap

The "Awesome Quality Assurance Roadmap" repository offers a structured path for QA engineers and software testers to enhance their skills. It provides a generic test plan sample, emphasizing the importance of effective test planning. Additionally, it shares expert advice, stressing the significance of understanding software testing fundamentals before diving into automation and broadening testing perspectives beyond server status codes. This resource equips testers with essential insights and resources to excel in their roles.

2. Free Learning Resources For Software Testers

The "Free Learning Resources For Software Testers" repository is an ongoing project that compiles free online learning materials for software testers, catering to both beginners and seasoned professionals. Recognizing the lack of dedicated software testing modules in computer science and IT departments, this project provides a structured curriculum and organized lists of relevant, non-promotional, and cost-free resources. While exceptions apply for paid resources like books and magazines, the repository serves as a valuable, ever-expanding resource for the testing community, encouraging contributions and offering a repository of free educational content.

3. Awesome Learning

The "Awesome Learning" repository documents the remarkable journey of a software testing professional who transitioned from manual testing to a Test Automation Engineer over 14 years. Starting as a Commerce graduate in 2008 with no engineering background, the author provides insights into the challenges of understanding programming, testing, and software releases. This repository serves as a guide for those aspiring to make a similar career shift, offering a structured learning path based on the author's experience and knowledge, inspiring others to excel in test automation.

4. SQA wiki

The "SQA wiki" repository is a collaborative platform that hosts a variety of resources focusing on software quality and testing including books in both English and Russian, learning courses in both languages, and tools that gamify the learning process. It encourages active participation, allowing contributors to edit existing articles and create new ones, fostering a community-driven knowledge base.

5. Awesome Testing Courses & Tutorials

“Awesome Testing Courses & Tutorials” provides A curated list of awesome online courses & tutorials on software testing, test automation and more.

6. Test Cases and Templates for Manual Software Testing

The "Test Cases and Templates for Manual Software Testing" repository is a valuable resource for manual testers seeking to streamline their testing processes. It offers a comprehensive set of sample test cases and templates for test planning and bug bashes, simplifying and accelerating the test case creation process.

The project's primary aim is to address the time-consuming nature of writing test cases in software testing. By providing generic test cases for commonly used functionality, such as text boxes, dropdown boxes, checkboxes, and more, testers can readily download and employ them in their projects, saving substantial time that would otherwise be spent on crafting generic test cases.

The repository includes sample test cases and templates for a range of testing scenarios, including export and import functionalities, testing reports, mobile apps, login functionality, and a test planning checklist. It serves as a practical resource for manual testers, enhancing efficiency and promoting best practices in the software testing domain.

7. Quality in Mobile Apps

The "Quality in Mobile Apps" repository is a one-stop destination for resources dedicated to mobile app testing, covering both iOS and Android platforms. It houses a diverse array of tools, guides, and references aimed at ensuring the high-quality performance of mobile applications. Whether you're a mobile app tester, developer, or quality assurance professional, this repository offers essential insights and tools to help you deliver top-notch mobile app experiences.

8. Performance Testing

The "Performance Testing" repository is a comprehensive resource for professionals looking to optimize software performance. It includes a wealth of tools and resources for both backend and frontend performance testing, as well as web performance optimization. This repository serves as a valuable guide to help individuals ensure their software applications run efficiently and deliver a seamless user experience.

9. How They Test

The "How They Test" repository is a curated collection of publicly available resources that offer insights into how software companies worldwide approach testing and build a culture of quality. It serves as a comprehensive knowledge hub, consolidating best practices, tools, techniques, and testing culture from various companies, all accessible in one place.

This repository covers a wide array of topics, extending beyond traditional functional testing and test automation to include non-functional testing, testing in continuous integration and continuous delivery (CI/CD), release management's impact on quality, quality processes, and even testing in production. Companies like Google, Netflix, and Amazon are among those sharing their valuable insights, making it an invaluable resource for those looking to learn and improve their software testing practices.

10. Software testing resource pack

The “Software testing resource pack” repository contains various files useful for manual testing and test automation to check REST API or webpage behaviour with various file extensions, sizes, resolution, names etc.

007-Inspired Challenge: Blind coding competition on James Bond Day

Natalia Demianenko — Thu, 12 Oct 2023 17:18:54 +0000

On October 5, it's not just a date on the calendar; it's an international spy holiday imbued with an aura of mystery and adventure. On this day, over half a century ago, the world first met James Bond — a symbol of skill, charisma, and incredible expertise. The James Bond franchise has captured the hearts of millions of viewers, immersing them in a whirlwind of captivating stories about spies, intrigue, and adventures.

At Devhouse, we regularly organize various educational and interactive meetups, as I mentioned in my previous article. On the occasion of this holiday, we decided to prepare a blindfolded web design competition. The concept is simple — participants see a layout and attempt to code it without seeing the result until the end of the competition.

To organize such an event in a distributed team, we needed to carefully plan the setup to make it engaging for both participants and observers. For this purpose, I prepared a repository that each participant cloned for the competition. Inside, there was a branch with a layout template (including basic sizes and some colors) and the necessary images. To share their coding progress with the organizer, updating in real-time, we utilized the browserSync library. It allowed each participant to host their index.html file at a URL like https://<my_custom_name>.loca.lt using the command npm start <my_custom_name>.

The next step was creating an interactive board to monitor the competition's progress. For this, I prepared a simple page with iframes corresponding to the number of participants, with links for each. I added functionality to hide/show participants' names (revealed only at the end, with a touch of espionage!). During the call, I simply shared this page in the browser with all the observers and participants.

Another detail to note is that by default, browserSync requires entering an external IP address to access someone else's result. However, this can be bypassed by modifying the user agent. This can be easily done in Google Chrome under the Network Conditions tab.

But what does James Bond have to do with all of this? I created a themed layout in Figma, and one of the additional tasks was to add animation to the "007" image, making it appear and disappear like a true spy.

At the end of the competition, we all came together to select the winners, considering the speed of coding, alignment accuracy with the layout (although a perfect pixel task wasn't the primary goal, we aimed for a fun and productive time), and creativity in animation development. See below for what we achieved. Which result is the winner, what do you think?

And there are some intermediate results, it was intresting to see the progress.

This experience at Devhouse shows that team-building in the IT sphere can be captivating, creative, and diverse. Perhaps your company will find inspiration in this experience and create something similar within your team.

Integrating Educational Activities: Insights from an IT Company

Natalia Demianenko — Sun, 17 Sep 2023 10:37:46 +0000

Hello everyone! We are Devhouse, a team of developers, testers, managers, and designers engaged in a dozen different projects. We have 35 team members, each with varying experience, skills, and grades, and we work entirely remotely across 10 countries. At the end of last year, we set ourselves a goal: to implement regular educational activities that would help everyone grow and develop, as well as facilitate communication not only within different project teams but also within the company as a whole. This falls under my area of responsibilities, so I'd like to share how we organized this.

We already had experience organizing internal meetups, which was quite positive, but for various reasons, it didn't become a stable practice. We drew conclusions from this experience, set new goals based on them, and gradually started to implement them.

Essentially, we had two major tasks at hand: motivating speakers and engaging the audience.

Engaging the audience

Before starting the meetup activities, we did a little warm-up by creating a knowledge library in Notion. Since we already managed all our documentation, handbooks, company information, employee details, and projects in Notion, the choice of the tool was straightforward. Everyone could contribute the most significant professional resources for them to the library—books, articles, YouTube channels, podcasts, and more—adding tags related to their professional areas for easy searching.

This way, we tested the waters regarding how willing people were to share knowledge, compiled an excellent collection of materials recommended for review, and allowed people to contribute to the collective development. The sources recommended by them are displayed on their Notion profile, enhancing interest and respect from colleagues.

We understood that we could move forward. The main ideas to generate and maintain interest in meetups were:

Choosing topics that are understandable to the majority, regardless of specialization and knowledge level
Gamification of the process

Choosing the Meetup Topics

Here we kept it simple - we asked the employees for their opinions. We conducted a survey listing topics related to various hard and soft skills and asked them to vote for the ones that interested them. This way, we identified topics that are definitely worth covering in the upcoming meetups and would be of interest to the majority. We also asked them to vote for the types of activities they were willing to participate in, such as games/competitions, collaborative coding, or watching presentations from major conferences. This helped us identify the most popular educational activities that we decided to try.

Gamification

For each meetup, we decided to include some interactive component, such as a quiz, activity in Miro, or questions from the speaker during the meetup. This allowed us to determine winners and the most active participants. After each meetup, in our corporate workspace on Slack, we congratulate the winners. Our designer prepared cool templates for award placements.

All of this allowed us to maintain a leaderboard in Notion, where at the end of the quarter, we can see who scored the most points. It turned out to be a good motivational system for participating in meetups.

The second aspect of gamification was quite challenging, but it was worth the effort. As mentioned above, we decided to organize games and competitions in coding, layout, and design as separate educational activities. For our first attempt, we chose an open-source game, coded the backend to process logins and leaderboards, and tweaked the frontend to display the results. We ended up with a super cool activity that marked the beginning of a series of regular meetups. Next, we replicated this approach for another game and plan to create several more similar activities.

Motivating the Speakers

Choosing the Topic

I myself am a frequent speaker and understand that the main motivation for giving a talk is your own development. Nothing hones and solidifies your knowledge like sharing it with others. However, this is why it's important for the speaker to already have a good understanding of the topic and not spend too much time preparing. Therefore, in the aforementioned survey, we also asked who was willing to give talks and what knowledge they were willing to share with colleagues. It's great that based on the results, we were able to find the first 5 speakers who were excited about the idea, fueled others with their enthusiasm, and helped make the process consistent.

Assistance in Preparation

However, public speaking and preparing presentations can be challenging for some. So, in the same Notion workspace, we created a preparation guide, describing the speaking protocol, providing a presentation template, offering preparation tips, instructions for creating a quiz, essentially everything needed to conduct a successful meetup.

This also streamlined management; instead of endlessly explaining these instructions to each speaker, we simply shared the link and only addressed any remaining questions.

Feedback

Any activity should have feedback. Therefore, after each meetup, we gather feedback, which helps the speakers understand how clear and structured their presentation was, what other aspects can be highlighted on the given topic, and what can be improved for the next presentation. In our team, we have a friendly atmosphere, and we are open to constructive feedback, which definitely makes subsequent meetups better.

Gamification

Yes, we didn't do without it here either. In our Notion workspace, we have a table for the most frequent speakers, and once a quarter, we highlight the most interesting presentations based on feedback. Naturally, appearing in announcements on Slack, being seen as an expert in a particular topic by colleagues, is quite pleasing.

Problems and and their solutions

Finding speakers

When the first stage of topic selection was completed, we decided to further automate the process and implemented the ability to propose new topics in Notion. We also included a voting feature for the most popular topics. However, we didn't get speaker suggestions from there, and from the voting, we extracted what was already clear and understood. The conclusion here is that having a person who periodically reminds about this and actively seeks speakers is crucial. For instance, noticing the most active participants in discussions during meetups or knowing precisely who is strong in a particular topic and can share expertise. The task of finding speakers is still challenging, especially when there aren't that many people in the team.

Games preparation

An even more challenging task is preparing educational game activities, which consumes a significant portion of my time. Partly because I'm not a programmer and coding is a hobby for me, and partly because it's inherently complex. But the success of each game is the main motivation to continue creating them.

Meetup topics

One of the main advantages - topics that interest the majority - is also a drawback. It imposes limitations on the depth and specificity of the presentation. However, considering that the main goals of the meetup are broadening horizons and team building, this choice is justified. For narrower topics, we can organize smaller events. For example, we have separate meetups for QA, project managers. We also communicate as needed to share experiences. In Slack, there are dedicated channels for developers based on different technologies, where you can seek advice from colleagues or share your own developments.

Meetup timing

Choosing the timing of the meetup is also a challenge when everyone should have the opportunity to participate, considering different time zones and their individual meeting schedules on projects. We tried to choose the optimal time and, of course, we record the sessions for those who missed the event.

Conclusion

To sum up, introducing consistent educational events in our IT company has been a beneficial initiative. Thoughtfully selecting universally understandable topics and adding playful elements have encouraged active participation and knowledge exchange within our diverse team. While some challenges remain, the positive results emphasize the importance of investing in ongoing learning and promoting team unity, especially in a remote work context.

Share, does your company have a practice of educational activities? I would appreciate your questions in the comments and your likes.

AI assistance in software testing: boost productivity and improve skills

Natalia Demianenko — Tue, 08 Aug 2023 21:39:00 +0000

In today's rapidly evolving software development landscape, testing plays an important role in ensuring the quality and reliability of applications. With the coming of Artificial Intelligence (AI), testing processes are being revolutionized by AI-powered assistants that enhance efficiency, accuracy, and speed. This article explores how AI helpers like ChatGPT, Postbot for Postman, and AI plugins for database tools like DBeaver and Studio3T are transforming software testing, making it better and more productive. I had a talk with my team last week about how we can use these tools especially for manual testing, and I want to share our ideas with everyone.

ChatGPT

There are numerous articles have highlighted the versatility of ChatGPT for software testing, e.g. for test ideas and test cases generation, requirements analysis, bug documentation and so on.

But this article is going to talk about something different and special about ChatGPT. It's really good at making different scripts. This is helpful for people who test software but might not know much about certain computer languages like JavaScript or bash. Based on your testing needs you can create script for generate text file with 1000 lines without knowledge of bash in 5 seconds. Moreover, ChatGPT will explain you how to create file for script and run it. Try prompt:

Create bash script to generate .txt file with 1000 lines

Another example of using ChatGPT for scripts generation is the following prompt:

Create script to run in browser console to get an array of urls from all images on the page

And again - 5 seconds and you are ready to get all you need with detailed explanation on how and what to do.

So beyond its practical applications, ChatGPT also serves as an educational tool, empowering testers to expand their skill set and experiment with automation.

OpenAI API

How can testers utilize the OpenAI API? I've previously covered the topic of generating realistic test data in an article, which involves a form of automation. However, it's important to note that manual testers can also benefit from this approach.

Postbot assistant in Postman

Once more, many testers might not be very familiar with coding. So, Postbot is an excellent tool that allows to write important tests which they might not have been able to do without the help of a developer or a lot of time doing research on the internet.

To start using it install the latest version of Postman. You'll find Postbot in the "Tests" tab of any request. To get it working, you just need to send the request once. This helps Postbot understand the response structure. After that, you can ask it to do specific tasks. It's a good idea to try it out while it's still in beta and free. Let's take a look at what this assistant can do.

Generate tests for requests

Simple prompt “generate tests for this request” will return up to 5 tests, you can improve prompt to generate less tests or provide details. For education purposes you can ask to add comments which explain what test do. It's amazing that tests generating duration is just a few seconds!

Visualize the response

Did you know about such functionality in Postman? It's pretty easy to create visualisation script with Postbot in a few seconds. Use prompt:

Visualize response showing the fields [fields]

And you will get the script to visualize your response in table format to make it easy to read.

Save variables from response

If you want to keep something from the response and use it later, you can use this prompt:

[Condition]- save [field] as [destination] variable

Postbot will quickly give you a script for that! It's pretty amazing how fast it works.

DBeaver AI plugin

I'm pretty sure your project involves using databases. If you're dealing with SQL databases, you might be using DBeaver to check things in your database. Most testers are familiar with SQL, but if you want to get better at it or speed up how you write queries, you can install the AI Smart Assistance plugin. Follow the instruction https://dbeaver.com/docs/wiki/AI-Smart-Assistance/. You'll need an OpenAI API key to make this work. Once you set it up, you'll see the chatGPT icon in the SQL script panel. Click on it and start typing your questions.

AI Helper for Studio3T

If your project involves MongoDB and you're looking for help with writing queries, the ChatGPT Plugin for Studio3T is what you need. To get started, download the latest version of Studio3T and enter your OpenAI API key: Go to Preferences -> IntelliShell -> OpenAI Api Key. After that, you can select the collections you want the assistant to work with and write your query in natural language. Remember, the assistant only knows the structure of your data and doesn't have access to the actual data itself. I really like the example of usage from official blog.

Conclusion

In conclusion, AI helpers like ChatGPT, Postbot AI, and AI plugins for database tools are revolutionizing software testing. These tools empower testers to automate tasks, generate scripts, and create realistic test data, enhancing efficiency and expanding skills. This collaborative synergy between AI and human expertise promises a future of more robust and proficient software testing practices.

Have you used these assistants? What are your thoughts on them? Share your experiences in the comments below. Also, let me know if you'd like to explore any specific aspects further. Your feedback and input are greatly appreciated!

Enhancing Bug Reports: How to Capture Full-Page Screenshots in different Browsers

Natalia Demianenko — Mon, 26 Jun 2023 19:53:01 +0000

Effective bug reports heavily rely on attachments, and in web testing occasionally arises a requirement to capture screenshots of complete web pages, not only the visible part. Luckily, popular desktop browsers offer diverse tools and extensions that can aid in this task. In the following sections, we will delve into the methods of capturing full-page screenshots across different browsers, ensuring comprehensive bug reporting.

Google chrome

Google Chrome offers a built-in command for capturing screenshots within its developer tools. Follow these steps to take a full-page screenshot:

Open Developer tools (by pressing F12).
Click the three-dot icon from the top-right corner and select Run command. Also, you can press Ctrl+Shift+P on Windows or Command+Shift+P on Mac.
Type screenshot into the search box.
Select Capture full-size screenshot from the options provided.
Once Chrome captures the screenshot, it should automatically save it to your Downloads folder.

However, if you find opening the developer tools every time to capture a screenshot inconvenient, you can opt to install extensions that simplify the process. There are several extensions available that allow you to capture full-page screenshots. Here are a few recommendations:

GoFullPage
Screen Capture

Some extensions also offer additional functionalities such as editing, drawing, and making notes on the screenshots. One extension that stands out for its comprehensive functionality is Awesome Screenshot, which provides a rich set of features to enhance your screenshot-taking experience.

Safari

Unlike some other browsers, Safari does not provide a built-in tool for capturing full-page screenshots. However, Mac OS does offer a built-in screenshot app that can be accessed by pressing Command-Shift-5. This app allows you to capture the entire screen, a selected area, or even record scrollable screenshots in video format. Nevertheless, if you specifically need to capture full-page screenshots in Safari, extensions are the way to go. One recommended extension for this purpose is Awesome Screenshot, which offers convenient and useful features.

Mozilla Firefox

Mozilla Firefox offers a convenient built-in solution for capturing full-page screenshots. To capture a full-page screenshot in Firefox, follow these steps:

Hold down the Control key and click on an empty part of the page to open the context menu.
Select Take Screenshot.
Choose Save full page.

Alternatively, you can use the keyboard shortcut Command + Shift + S or pin this tool in browser panel.

The built-in screenshot tool in Firefox also allows you to capture visible portions or selected areas of the page, eliminating the need for additional extensions. However, if you require features such as making quick notes or drawings on the screenshot, you can install add-ons. One recommended add-on for this purpose is Awesome Screenshot.

Microsoft Edge

Microsoft Edge provides a built-in tool for capturing web pages. To take a full-page screenshot in Microsoft Edge, follow these steps:

Click the Menu (...) icon located in the top-right corner.
Select the Web capture option.
Click the Full page option to capture the entire webpage.

You can also use the shortcut Ctrl + Shift + S to quickly bring up the Web capture tool.

Opera

Opera also has useful built-in solution for taking full-page screenshots. Simply click on the photo camera icon located on the right side of the search bar and select Full Page Screenshot.

Once again, if you require annotation functionality, you can install an add-on such as Droplr - Screenshot, Annotate, & Screencast.

Conclusion

In conclusion, capturing full-page screenshots in different browsers is made possible through the utilization of built-in features and add-ons. In addition to built-in tools, installing extensions can provide convenient options for annotation and additional functionality. You are welcome to share in the comments which browser and tools you use for everyday testing and also provide any recommendations for other convenient options or add-ons that you find useful.

How to prepare realistic test data via OpenAI API in Postman

Natalia Demianenko — Sun, 18 Jun 2023 14:18:11 +0000

How to generate diverse test data? And what if you need a realistic dataset for a product demonstration? Test data generation can be a challenging task. However, with the advent of AI, it has become much easier. This article demonstrates how to generate test data using the OpenAI API in Postman and automatically send it to your server. But let's start with an introduction.

If we have an environment designed specifically for testing, the data often consists of identical values with the word "test". It is quite difficult to navigate through such a set. We can use random data generators, but in this case, we lose the relevance to the subject area of the tested product. The realism of test data is especially important for a product demo. Manual creation of test data is an option, but if the schema of one data object is complex and a large number of objects are needed, it becomes time consuming and impractical.

So, how to generate realistic and diverse test data without high time costs? To obtain such result AI generation can be used. Let's assume we have a web application with a catalog of the most popular laptops. And in our test environment, the products themselves don't exist yet, so we need to create them. To do this, we will use the public API https://restful-api.dev/ which provides the ability to write our data via POST request in the following format:

{
   "name": String,
   "data": Object
}

That is, according to the API documentation, the structure should contain a name and any data as an object. Let’s see how to create test data for our application in a few simple steps.

Step 1. Determine test data object structure

Let each product in our application contain the following data:

{
   "name": "Apple MacBook Pro 16",
   "data": {
      "year": 2019,
      "price": 1849.99,
      "currency": "USD"
      "CPU model": "Intel Core i9",
      "Hard disk size": "1 TB"
   }
}

So the structure of the items we should generate is the following:

{
   "name": "String",
   "data": {
      "year": "Number",
      "price": "Number",
      "currency": "String",
      "CPU model": "String",
      "Hard disk size": "String"
   }
}

Agreed, manually creating a dozen different and realistic items for subsequent testing can be challenging. This is where the OpenAI API comes to our aid.

Let's create a collection called "Create Test Data" and the first request called "Generate Test Data". In Pre-request script let's define JSON structure (to be able to easily change it before converting to string) and store it as a collection variable.

Step 2. Generate test data object via OpenAI API

To generate the items, we need to send a request to the following endpoint: https://api.openai.com/v1/chat/completions, which allows us to obtain a response for the given chat conversation. Let's construct the body of the request to the OpenAI API.

In the content field, we specify what we want to receive from the model in response, with the exact expected JSON structure defined on the Step 1.

The temperature parameter allows us to adjust the predictability of the model's responses. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. It accepts values from 0 to 2.

If the API of your application can accept an array of products for creation, you can add the n parameter, which allows you to configure the number of responses. Another way to generate a dataset is to change request on something like: Generate array (length 2) of objects describing a laptop items using the following structure… The way to generate a set affects on how to extract the information for the next request.

Since our test API only accepts one object at a time, we use the request for single object generation and do not pass the n parameter, and by default, it is set to 1.

To try the example you can import the following curl to your Postman workspace.

curl --location 'https://api.openai.com/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $OPENAI_API_KEY' \
--data '{
    "model": "gpt-3.5-turbo",
    "messages": [
        {
            "role": "user",
            "content": "Generate object describing a single laptop item using the following JSON structure \"{\"name\": String,\"data\": {\"year\": Number,\"price\": Number,\"currency\": String,\"CPU model\": String,\"Hard disk size\": String}}\""
        }
    ],
    "temperature": 0.7
}'

OPENAI_API_KEY is your authentication key. Please visit your API Keys page to retrieve the API key you will use for your request.

You can play with the parameters to make the result suitable for your purposes. See OpenAI API documentation for more information.

The response to our request will be in the following:

{
    "id": "chatcmpl-7SB7rzphTMzpB9VrLDtUknZyPQjez",
    "object": "chat.completion",
    "created": 1686950307,
    "model": "gpt-3.5-turbo-0301",
    "usage": {
        "prompt_tokens": 51,
        "completion_tokens": 70,
        "total_tokens": 121
    },
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "{\n  \"name\": \"Dell Inspiron 15\",\n  \"data\": {\n    \"year\": 2021,\n    \"price\": 800,\n    \"currency\": \"USD\",\n    \"CPU model\": \"Intel Core i5-1135G7\",\n    \"Hard disk size\": \"512 GB SSD\"\n  }\n}"
            },
            "finish_reason": "stop",
            "index": 0
        }
    ]
}

So, in the content field, we receive the test data that we wanted to generate - realistic and matching our structure. Now, let's store them in a collection variable for subsequent sending in the body of the next request.

Step 3. Store generated data as collection variable

To do this, in the "Test" tab of "Generate Test Data" request, we will write the following script:

var jsonData = pm.response.json();
const item = jsonData.choices[0].message.content;
pm.collectionVariables.set("item", item);

Now, after sending the request, the script will execute and store the received response in the collection variable named "item".

The test data is ready, and it's time to use them.

Step 4. Send data to your server

Let's create another request in the collection called "Send Data".

curl --location 'https://api.restful-api.dev/objects' \
--header 'Content-Type: application/json' 
}'

As the request body, we will send the JSON that was saved in the collection variable.

The collection is ready. To generate multiple products, let's setup the configuration to run multiple iterations.

As a result, our application now has various real test data. Quickly and easily!
Please share in the comments how do you utilize the OpenAI API for testing purposes?

Efficient Visual Layout Testing with Chrome DevTools

Natalia Demianenko — Fri, 09 Jun 2023 21:24:36 +0000

A web tester, in addition to functional testing, faces various non-functional checks such as UI testing, usability testing, performance testing, etc. One important aspect of web interface testing is checking the layout of web pages. In this case, we not only check the appearance of the page, but also its compliance with the design, display of animations, and other aspects that may affect the user experience.

Of course, every tester is familiar with developer tools and applies them in their work. However, not everyone knows that developer tools can provide additional features that can significantly simplify testing. In this article, we will focus on Chrome developer tools and explore some handy tricks that will help you easily check the layout and obtain more accurate results.

Colors testing

How can we verify if the colors on a web page match the style guide? How can we identify colors that don't respect to the design? Of course, you can rely on your eyes, you can inspect each element and look at its CSS properties, but there is an easier way to discover all the colors used on the page - the CSS Overview panel. This is a new feature still in preview mode, which allows you to test it as a user and provide feedback.

You can access the panel in two ways:

Through the kebab menu (Customize and control DevTools) → More tools → Css overview
Through the Command menu → Show CSS Overview

In addition to other information, the panel provides data on all the colours on the page. The colors are grouped by type (background, text color, border color, etc.). Clicking on any color reveals a list of elements where it is used. Hovering over an element from the list highlights it on the page. There is also information about low-contrast combinations, which is important for accessibility testing.

When inspecting individual elements, we can see that the color is represented in different formats - HEX, RGB, HSL. To quickly obtain the desired format for verification, hold down the Shift key and click on the color icon. As a result, you will get a list of color representations in different formats - quickly and easily without additional clicks to open the color panel.

Fonts testing

Another challenge during web pages testing is how to make sure that all fonts match the design? The easiest way to accomplish this is to use CSS Overview panel once again. This panel also provides information about all the fonts used on the page. This makes it easy to identify fonts that doesn’t adhere to the style guide, determine which elements are using, and create bug reports based on this information. The functionality is the same - by clicking on the font, we get the elements with it, and by hovering over the element, the corresponding element is highlighted on the page.

Pop-ups testing

One of the common problems in visual testing is how to test pop-up elements that are dynamically rendered on the page. Especially if there is a bug in them and you need to document it by attaching a screenshot. There are two methods to freeze such elements for testing. The first method involves using a special debugger on the parent element.

If you find it difficult to determine exactly where in the tree the element appears, you can set a breakpoint on the modification of the entire

. This way, when the pop-up elements are triggered, the execution of scripts responsible for their rendering will be paused, allowing you to step through until the desired element appears on the page. However, this method can be complex for testers due to the challenges of determining the parent element and the unpredictability of the number of debug steps required to achieve the desired result.

The second method is to set a debugger after a certain amount of time, during which you need to call the pop-up window. Simply type the following command in the console:

 setTimeout('debugger', 5000)

press Enter, manage to call the pop-up element within 5 seconds (5000ms) and test it calmly. I personally prefer this method, as it is faster and more convenient.

Please note that this method may not work on native popup elements such as date pickers and dropdowns. Beginners may sometimes have doubts about how to recognize native browser elements and why they are not visible in the regular DOM. To ensure that you see the native element you can go to Settings → Preferences → Show user agent shadow DOM. This will enable you to see and inspect such elements.

What is important for tester in terms of user agent shadow DOM? It's essential to understand that the behavior and appearance of the User Agent Shadow DOM elements are controlled by the browser vendor. While developers can modify the styles of user agent shadow DOM elements to some extent using CSS, it is important to note that any changes made may have unpredictable effects and may not be supported across different browsers. So in most cases testers should not report bugs or issues related to the encapsulated styles or behavior of these elements to their development team, as they have limited control over them.

Animations Testing

When testing layouts, we occasionally come across animations. How can we ensure that animations play correctly according to the design? TDeveloper tools offer a specialized tool for debugging animations. The most convenient feature for a tester here is the ability to slow down animations, allowing careful observation of their progress to catch any bugs. Additionally, it's possible to pause an animation to report any issues encountered.

Responsiveness Testing

As you may have noticed CSS Overview contains more than just fonts and colors information. Another essential piece of data that testers can obtain from this panel is the list of media queries. What is the media query and why it’s important knowledge for tester? A media query is a CSS rule that allows you to control the styles of elements based on the values of technical device parameters, such as viewport width which is fundamental measurement for responsiveness testing.

From here, we can get the boundary values of viewports that need to be included in responsiveness testing. By applying test design techniques such as equivalence classes and boundary values to screen width values in media queries, the number of devices required for checking layout responsiveness can be reduced.

The same information is also available during device emulation. In additional settings in emulation mode, you can enable the 'Show media queries' option. This will display a panel showing the boundaries of screen widths for which specific styles are applied.

Bonus! Useful Chrome extension for Responsiveness Testing

Dynamic content testing

Sometimes, while testing, there is a need to check the layout with different text to ensure that elements dynamically change based on the content or that the layout remains stable on mobile devices. We can prepare proper test data for this, or we can simply find the necessary element in the DOM tree using the inspector and replace the text within it. However, if there are multiple elements requiring changes, it is more expedient to switch to design mode. To do this, type the following command in the console:

document.designMode = 'on'

Once activated, any text on the page can be directly edited.

Screenshots for Testing Reports

No layout testing report would be complete without screenshots. Of course, you can use some extensions or separate programs, but in Chrome developer tools there is a ready-made functionality for all types of screenshots. You can capture the visible area of the browser window, take full-screen screenshots, select a specific area, or even capture individual elements. To access this feature, open the command menu, start typing “screenshot” and select the desired item from the list.

Additionally, there is an alternative method for capturing screenshots of elements. Right-click on the node you want to capture and select the corresponding menu item. Save the screenshot and attach it to your bug report. It's quick and easy!

Conclusion

One of the most valuable soft skills of a tester is curiosity. Apply it to everything you encounter in your work. Open up DevTools and explore its features, try out different functionalities, and familiarize yourself with the available commands. Dive into the settings and customize DevTools to suit your daily tasks. Learn keyboard shortcuts to speed up your work. Don't overlook articles about Chrome updates. You may not understand everything mentioned in those articles, but there's a good chance you'll find something useful and convenient for your work purposes.