DEV Community: DenisC

Review on My Work on Scrapy

DenisC — Sat, 13 Dec 2025 04:57:16 +0000

In my previous contribution to Scrapy, I worked on code cleanup for test configuration as a starting point to familiarize myself with the project structure and development workflow. That work helped me understand how Scrapy organizes its test setup and how changes are validated through its testing framework.

This time, I moved a step closer to Scrapy’s core data extraction logic by working on a refactoring task related to how spiders are configured. Compared to test-related cleanup, this task required a deeper understanding of how Scrapy’s internal components interact during crawling and configuration.

Digging into Interrelated Issues

The main goal of the issue was to deprecate an old attribute previously used to configure spiders. This attribute has been replaced by a list of settings, which provides a more consistent and unified configuration flow across the project.

To work effectively on this task, I had to explore earlier issues and pull requests that introduced or discussed the change. Reading through the discussions and code changes helped me understand the motivation behind the refactor, the design decisions that were already made, and the constraints I needed to respect.

This experience showed me a common pattern in large projects that changes are often interrelated. Features and refactors are often split across multiple issues and pull requests, meaning contributors must spend significant time studying project history. While this increases the initial learning curve, it also ensures that changes remain consistent and well thought out.

Working on the Issue

After studying the issue and related discussions, I realized that the refactor involved more than just modifying class definitions. I also needed to update existing tests so that they validated the new settings list instead of relying on the deprecated attribute.

To make my approach clear, I created a draft pull request outlining my plan and progress. This made my work transparent and allowed others to follow my reasoning early on.

During implementation, I relied on tools like git grep to locate all relevant references to the deprecated attribute. This ensured that the refactor was thorough and consistent across the codebase. After completing the changes, I updated the pull request summary to explain what was changed, making it ready for review.

After receiving the maintainer's comment, I further explained the rationale of changes and the logical flow of the updated codes. The maintainer later suggested that it would be better if another issue is tackled first, so the pull request is kept open at this stage. Although the pull request has not been merged, the work I've done and the discussion may provide some insight to other contributors, or even my future self.

Reflection on My Work on Scrapy

Through this contribution, I’ve become more comfortable working with large and complex codebases. I’ve learned that progress in such projects is incremental, as the saying goes, Rome wasn’t built in a day. Understanding comes gradually through repeated exposure, patience, and continuous effort.

This experience also reinforced the importance of refactoring and cleanup. These maintenance tasks are essential for long-term project sustainability, especially as project requirements evolve and programming languages introduce new features or deprecate old ones. Without careful maintenance, projects risk being overwhelmed by technical debt or falling victim to problems like the second-system effect.

Looking back at my initial expectations, I’m glad that I was able to improve Scrapy’s code readability and maintainability. At the same time, I now better understand that fully grasping a project of this size is a long-term process. I plan to continue contributing to Scrapy and using it in the future to deepen my understanding further.

Review of My Open Source Journey

Since September this year, I’ve been actively involved in the open source community. My journey started with working on my own projects and collaborating with my classmate, then gradually expanded to contributing to public open source repositories on GitHub.

I began with smaller contributions such as bug fixes, adding features, and writing tests, which helped me understand contribution workflows and project structures. Over time, I took on more involved tasks like deprecating older Python version, optimizing test, and eventually refactoring code in larger project.

Working on open source has consistently challenged me to step outside my comfort zone. By progressing from small fixes to more complex refactors, I can clearly see my growth in both programming and problem-solving skills.

Open source projects are a valuable learning resource, especially for beginners. They provide opportunities to learn from experienced developers across diverse domains. I plan to continue my open source journey, sharpen my technical skills, and build meaningful connections within the community. Looking forward to contributing to impactful projects in the future!

Progress Updates on Contribution to Scrapy

DenisC — Thu, 11 Dec 2025 05:46:02 +0000

Last week, I worked on code refactoring in Scrapy, which is an essential practice in larger and more complex projects. Refactoring not only improves code maintainability but also makes it easier for other contributors to understand and extend the project. This task was a good starting point for me to verify that I had the Scrapy project correctly set up locally, as refactoring of codes should not break existing functionalities.

About the Issue

The issue I worked on is #7141, which aimed to refactor the multiple autouse fixtures used for pytest.skip() in conftest.py into a single pytest hook. The original code had multiple small fixtures that checked for specific markers and optional dependencies, which made the code harder to maintain and less readable.

What I Have Done

To tackle this issue, I first studied the use of autouse fixtures and pytest hooks in Python, which I wasn’t very familiar with before. I learned how pytest hooks could centralize repetitive logic and how they interact with test markers and configuration options.

I then refactored the multiple autouse fixtures into a single pytest_runtest_setup hook, which checks for reactor-specific markers and optional dependencies. Instead of importing modules and deleting them as in the original code, I used importlib.import_module() to dynamically import modules only when needed to make the codes more concise.

To ensure my changes did not break existing tests, I used tox to run the test suite locally, before eventually submitted my pull request.

Lessons Learnt and Future Plan

This task reinforced the importance of code cleanup, especially in large projects, to reduce technical debt and improve readability and maintainability. I also learnt the significance of reading and following the contribution guide, which may vary between repos. For example, in Scrapy, concise commit messages and installing pre-commit hooks before submitting a pull request are essential practices. I noticed that some contributors overlook these guidelines, and maintainers frequently remind them, which highlighted the importance of carefully following project norms.

Looking ahead, I realize that understanding the full crawling process in Scrapy is more complex and time-consuming than I initially thought, but I am eager to continue contributing. My goal is to work on issues related to the core crawling logic, allowing me to gradually deepen my understanding of the framework.

Contributing to Larger Open Source Project - Scrapy

DenisC — Sun, 07 Dec 2025 03:51:30 +0000

In the past three months, I worked on various open source projects, including my own project Repo Context Packager, Math Worksheet Generator and Open Web Calendar. This month, I want to challenge myself to work on a larger and more widely used project - Scrapy, a Python module for web crawling.

Why I chose Scrapy

By working on several issues in Open Web Calendar, I've gained experience in working on Python project with a comprehensive test suite and continuous integration. I want to push myself further so I tried to find a larger Python project to work on. Besides, I want to find a project that I would use on a regular basis. One of the topics I'm interested in is web crawling, and I do need to extract data from online sources sometimes for statistical analysis. I then searched for "open source web scraper", and found Scrapy, which happens to be a great choice that satisfies all the criteria, with a large user base, plenty of issues to work on, and a well organized program structure.

My Work Plan

I will begin by carefully reading Scrapy’s official documentation and contribution guidelines to understand its core concepts, project structure, and coding standards. Next, I will install Scrapy locally and experiment with building a few small crawling projects, which will help me better understand how Scrapy components work together in practice. After gaining familiarity with both the documentation and actual usage, I will start exploring the issues on Scrapy’s GitHub repository. Since there are hundreds of open issues, I should be able to pick some issues that I'm interested in. Eventually, I should be able to submit pull requests following Scrapy’s contribution process and improve my work based on feedback from maintainers.

Expected Outcomes

Through contributing to Scrapy, I expect to gain a deeper understanding of web crawling and data extraction, by learning how professional programmers design crawlers to handle data extraction efficiently. Another important outcome is being able to directly improve Scrapy itself, By fixing bugs, enhancing features, or improving documentation, I can contribute to a tool that is widely used by developers around the world, which is both meaningful and motivating. Finally, I aim to become a long-term Scrapy user. After becoming comfortable with the framework, I plan to use it for real data extraction tasks related to my own research and statistical analysis, making it a core tool in my future projects.

Packaging my Open Source Project - Release 1.0.0

DenisC — Mon, 24 Nov 2025 06:43:52 +0000

This week, I finally published my project to PyPI. I relied heavily on the official Python packaging tutorial, using hatchling as the build backend and publishing through PyPI. Following the recommended modern workflow, I generated the distribution archives using python3 -m build and uploaded them with twine, after creating a PyPI account and configuring an API token.

Before any of that could happen, I needed to restructure my project to match the standard packaging format. This meant placing the actual source code inside a src/ directory and creating another package folder inside it, renamed as src/repo_context_packager. I also ensured the project root contained essential files such as LICENSE and README.md, and I created a pyproject.toml file to define the build system and project metadata. With these pieces in place, I was able to build the wheel and source distribution and prepare them for upload.

The most frustrating part of the process came from re-organizing the file structure. Moving the package from src/ directly into src/repo_context_packager meant that several parts of the project had to be updated. All internal imports needed to reference the new path, test files that depended on relative imports had to be adjusted, and usage instructions in the README required rewrites as well. Even GitHub workflow files were affected because they referenced specific paths for testing and packaging. Although the changes were not complicated, together they required careful attention to ensure nothing broke.

After the initial release, I also received useful feedback from early users. Someone pointed out that I had misspelled requirements.txt as requirement.txt in my installation command, which I quickly corrected. Another issue was that I had forgotten to update a path in the usage instructions after moving my files. The README originally told users to copy a template TOML file into src/, but the correct destination after restructuring was src/repo_context_packager/. I updated the documentation to reflect this so new users would not run into errors.

If you want to try the package, there are several easy ways to access it. It can be downloaded directly from PyPI page, installed via the command pip install repo-context-packager==1.0.0, or obtained by cloning the GitHub repository. Once the package is installed or downloaded, you can follow the instructions in the README to get started. Feel free to file an issue if you find any bugs or have any suggestion!

Contributing to Optimization in Open Source Project

DenisC — Thu, 20 Nov 2025 05:09:57 +0000

I continue working on Open Web Calendar after I resolved an issue to deprecate Python 3.9. This time, I try to work on another type of issue that I've never done before - optimization of documentation build for testing.

About the issue

The issue is to add a quick documentation build for testing, by introducing a MkDocs configuration file (mkdocs.test.yml) that inherits from the main mkdocs.yml. Using this configuration file, documentations should be built in only English and one more well-translated language using MkDocs, unlike building documentations in full set of languages as specified in mkdocs.yml. The quick documentation build should be incorporated in a new tox test, which will be integrated into GitHub Action specified in run-tests.yml.

The progress of problem solving

As I have not handled similar issues before, I didn't know how much time I would spend on this issue, but I wanted to let the maintainer and the community know that I was working on the issue and show my progress. This was the perfect scenario to create a draft pull request, to allow the community to check my work in progress while preventing accidental merges. I made the draft pull request after pushing the mkdocs.test.yml file, indicating the work done and to be done, as shown in the image below.

The remaining work was to update tox.ini and .github/workflows/run-test.yml for the new documentation build as suggested by the maintainer in the issue. Given my limited knowledge to tox and Github workflow, I had to do some research on how to change the files. I also found it useful to make reference to existing codes, not only could I make my codes more consistent, the codes also provided templates and comments that indicate the subtle details I should beware of. Making reference to existing codes while researching along the way, I was able to add the refined documentation build to tox.ini and run-test.yml.

Errors often pop up in the progress of solving an issue, this time is no exception. The first error I encountered was the failure to build the documentations with the new tox command, but thanks to the detailed logging from tox, I was able to locate the incompatible dependency that blocks the build, and fix it accordingly. Everything seemed to run well, I pushed my codes and they passed the check by GitHub Actions, including the new documentation build test I added in run-tests.yml, but something was still bugging me...

The documentation build I added was way quicker than the original build, the full documentation build for all languages (30+) took 24 mins, but my new documentation build for 2 languages only took around 10 seconds, which was disproportionate, and the pages were not properly translated. The tox command and GitHub Action worked fine, so I suspected that the mkdocs.test.yml, which I created at the beginning, was the source of problem.

Luckily, I got a hint from the maintainer, who provided a link about configuration inheritance in the issue he posted. It turned out that I had to use the key/value pair syntax for the plugins option in both parent and child configuration file, in order to override the values, which were the language list in this case. I thereby made changes to both mkdocs.test.yml and mkdocs.yml, tested the new documentation build in local machine, and got the translated pages in 2 ~ 3 mins build time, which was proportional to the original build.

Eventually, I pushed all my changes, passed the GitHub Actions check, edited the summary of changes, and changed my pull request to "ready for review" for the maintainer's acceptance.

Lessons learnt

First of all, I have learnt the importance of making the work transparent to the open source community, even if the work is still in progress. This prevent the case of different people working on the same issue redundantly, while allowing others to know the progress and check the work in case things go wrong.

Moreover, I realized that there was no single best way to find solutions. We often have to find different sources, such as the official documentation, existing code references, and discussion on forum, to cross-check the finding and validate the solutions, as the answers from different sources may be different and sometimes mistaken.

After working on this and my last issue on Open Web Calendar, I have known more about the file structure, general workflow, and maintenance work of open source projects, and gained confidence in solving unfamiliar issues while acquiring new set of knowledge in the future. Looking forward to solving new issues in the next few weeks!

Continued Journey on Open Source Projects - Deprecate Python 3.9

DenisC — Mon, 17 Nov 2025 02:05:39 +0000

My journey on open source projects doesn't stop after making four pull requests for Hacktoberfest last month. Contrary to finding different repos to work on different issues in Hacktoberfest, I'm focusing on one repo this time, namely Open Web Calendar. It is a well organized and maintained repo I worked on in Hacktoberfest, with rigorous tests and Continuous Integration(CI).

The issue I worked on

The issue I worked on is to deprecate Python 3.9, which reaches its End of Life in Oct 2025. The maintainer posted this issue giving a direction of what need to be changed, and mentioning the rationale for this issue, which is to remove Python 3.9 form tests in order to update dependencies which has deprecate Python 3.9.

Challenges I faced

The first challenge I faced was to understand the project structure. The issue is different from the issues I previously worked on, such as bug fixing and implementing new features, which allowed me to only focus on a part of the codes. This issue instead requires understanding the structure of the whole projects, then searching for relevant files and making changes accordingly. To get a better understanding on the repo, I used Repomix, a tool I studied in previously lab, to pack the info of the repo into an .xml file and send it to AI to get a brief summary of the project structure.

After understanding the project structure, I had to figure out which files should be changed. Based on the brief explanation from the maintainer, I did some research on how to handle deprecating a programming language version in open source project, and was able to narrow down the scope of changes to the following files:

documentation which mention the use of Python 3.9, located in docs/dev/
yml files in .github/workflows, which controls the CI process
tox.ini, configuration file for tox, the tool used in Python testing
pyproject.toml, configuration file for the project

The most tedious process was to find which part of the files should be changed. I used different approaches to do it, which include checking relevant files one by one manually, and use commands such as git grep py39. I had to pay careful attentions to spot out and remove Python 3.9 in the files, as Python 3.9 might appear in different forms, such as py39, python: ["3.9", ...], and python3.9. Moreover, as the maintainer intended to just remove the tests for Python 3.9 while still allowing users to install the app with Python 3.9, I cannot blindly erase all py39 syntax. For example, requires-python = ">=3.9" in pyproject.toml was kept.

After checking all the .yml in .github/workflows/, .md in docs/dev/, tox.ini and pyproject.toml and making relevant changes, I committed those changes, but new problems arose. As ruff now is run based on Python 3.10 instead of 3.9, syntax in some of the .py files need to be changed. Luckily, the logs from ruff clearly gave hints to fix the syntax, so I changed the syntax accordingly, pushed the codes, and made a pull request, which was eventually accepted by the maintainer.

Lessons learnt

First, I've learnt the importance of utilizing tools, which can save a lot of time in software development. From automation tools such as GitHub Actions and tox, to linter such as ruff, an even Repomix I used in the process of solving the issues, all these tools smoothen the software development process in different aspects, from running tests, fixing syntax, and solving an issue.
Moreover, I have become more confident in solving an issues before I have full knowledge about the issues. Unprecedented issues may arise when we are working on an issue, just like how the ruff check failed at the first time, but clues will appear in the progress, and community such as the maintainer and other developers will offer a helping hand. Finally, I now better appreciate the importance of clear logging. On one hand, the logging from ruff guided me in fixing the syntax. On the other hand, my efforts in writing clear commit messages and summary of changes were acknowledged by the maintainer, who could therefore quickly understand and approve the changes.

Setting Up GitHub Actions CI Workflow for My Open Source Project

DenisC — Thu, 13 Nov 2025 06:20:46 +0000

After writing tests for my open source project last week, I've set up CI Workflow with GitHub Actions this week, to better protect my repo and automate the testing process.

Creating the Workflow

I set up a GitHub Actions CI workflow by navigating to the “Actions” tab on my repository page in GitHub. From there, I selected a Python application workflow template, which gave me a starting point for automating testing my code.

The workflow is defined in a YAML file located at .github/workflows/python-app.yml, which includes steps like checking out the repository, setting up Python, installing dependencies, linting the code, and running tests. The file configures the workflow to automatically run whenever code is pushed to the main branch or a pull request targets it, ensuring that the codes are linted and tested consistently.

Challenges When Creating the Workflow

The workflow didn’t pass at first due to some file naming and structure conventions. For instance, my requirements.txt file was originally named requirement.txt (missing the "s"), so the line: if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
could not find the file. Renaming the file name to requirements.txt fixed the issue.

Similarly, the tests were initially in a single file called test.py. Pytest could not detect this file automatically. At first, I changed the workflow step to: pytest src/test.py -v to make it work. However, I wanted to follow best practices, so I moved the tests into a tests/ directory and split them into test_content_writer.py and test_util.py. Moving the tests also required setting the PYTHONPATH environment variable in the workflow:

env:
  PYTHONPATH: ${{ github.workspace }}/src

This allowed the tests to access the code in the src/ folder. With these changes, the workflow passed using the simple pytest -v command, and the project now follows proper Python conventions.

Writing Tests for Others

Beside setting up CI workflow for my repo, I also wrote tests for my classmate’s repo, which introduced additional challenges. The project's modules were organized differently from mine, so I had to carefully read through the codes to understand how different components interacted before writing the tests. For instance, I used unittest in my project, while my classmate used pytest module when writing the test, and there were subtle differences in the test files. I also had to study the logic of the codes carefully before writing the tests, to ensure that the tests correctly check for the function's intent.

Lessons learnt from CI Workflow

I have learnt that CI workflow is a great way to provide automatic testing, where every push and pull request triggers tests to catch errors early. It's great that it gives developers immediate notifications when the workflow breaks, just like how I received the emails about failure of PR run when I was setting up the workflow. It can also promotes better team collaboration by using standardized workflows that reduce integration issues across multiple contributors.

Writing Unit Tests for My Open Source Project

DenisC — Mon, 10 Nov 2025 06:10:04 +0000

Writing unit tests is an essential practice for ensuring code quality, reliability, and maintainability. This week, I wrote test cases for my open source project in Python.

Why I chose `unittest` framework

I chose unittest framework because it is included in Python standard library, which means there is no need to install external dependencies. It provides a structured way to write tests using classes, and a variety of assertions, which makes it easier to compare expected and actual results. Moreover, unittest has a well established community and documentation, making it easy to learn and find examples when needed.

How I set up the tests

To set up testing in my project, I first created a test.py file to keep all my test cases organized. I imported the necessary unittest module, and other modules when writing the tests, such as tempfile, os, time, Path from pathlib, as well as the modules from my project, namely util and ContentWriter. I also organized the tests into two classes:

TestUtil, for testing utility functions in util.py
TestContentWriter, for testing methods in ContentWriter in content_writer.py

Then, I wrote individual test methods for each function, to include the valid, invalid, and edge cases. Once the tests were written, I could run them all at once with the command python3 -m unittest test.py -v, which provided a clear report of which tests passed or failed.

Difficulties writing the tests

I initially struggled with finding the right modules and functions to use in my tests. For instance, I had to research the tempfile module to create temporary files and directories safely. I also learned how to use os.utime() to modify file timestamps in order to test the is_recently_modified() function. These challenges show the importance of searching the right tools in testing, not only the testing framework, but also modules that are useful in testing.

Lessons learnt

Overall, the process of writing tests demonstrates the value of testing in software development. It forced me to think about edge cases I might have overlooked and gave me confidence that my code behaves as expected. Using unittest is straightforward and powerful enough for the needs of this project. From this experience, I have learned that testing should be an essential part of every project, and I plan to study how to automate the testing process in future projects to maintain code quality and catch bugs early.

Hacktoberfest Recap - A Month of Open Source Learning

DenisC — Thu, 30 Oct 2025 21:34:11 +0000

After a month of coding, debugging, learning, and contributing, Hacktoberfest has finally come to an end. Throughout the month, I made four pull requests across different repositories, each focusing on a different aspect of open-source contribution, from bug fixing to adding new features and writing tests. Looking back, this month has not only improved my technical skills, but also deepened my understanding of collaboration and project management in open source.

Exploring and Getting Started

In the beginning, the biggest challenge wasn’t the coding itself but finding the right repository to contribute to. With countless projects labeled #Hacktoberfest, I learned to filter by activity and relevance, by looking for well-maintained projects where my contribution would be meaningful. My first pull request was for a Notepad web app, where I improved the title case conversion function. This first experience gave me confidence and a sense of accomplishment, proving that even small improvements can make a difference.

Diving Deeper into Bug Fixing

In my second week, I shifted my focus to fixing a reported issue in the Open Web Calendar project. This time, the work required setting up a complex Python development environment and understanding the project’s structure. Although only a few lines of code were changed, the process taught me the importance of documentation and testing. I realized that debugging isn’t just about finding errors, it’s about understanding how all parts of the system interact.

Adding a New Feature

The third pull request was both challenging and rewarding. I contributed to a Math Worksheet Generator by adding a new front page with a date, name, and score section, plus an optional title feature. To do this, I had to learn how to use the fpdf library from scratch. This experience taught me that we don’t need to master every tool entirely, focusing on the relevant parts is often enough to create something valuable. It was also my first time proposing and implementing a feature that wasn’t previously assigned, showing me how initiative can lead to meaningful contributions.

Writing Tests

For my final pull request, I focused on writing a new test case for the same Math Worksheet Generator project. This involved identifying missing test coverage, ensuring random question generation worked properly, and learning to maintain consistent coding style. I also learned a valuable lesson about Git branching after mistakenly branching off the wrong branch, I had to redo part of the work. This experience shows the importance of version control awareness when working on multiple contributions.

Final Reflection

Hacktoberfest has been a month of growth, patience, and discovery. I learned that contributing to open source isn’t only about writing code, but also about communication, organization, and adaptation to each project's own tools and ways of thinking.

Although Hacktoberfest has ended, this experience has motivated me to continue contributing to open source in the future. Whether it’s fixing bugs, improving documentation, or writing tests, every contribution counts and brings me a step closer to becoming a better programmer.

Hacktoberfest Last Pull Request - Writing Test

DenisC — Thu, 30 Oct 2025 21:31:41 +0000

The last pull request that I worked on was to add a test to Math Worksheet Generator, which is posted in this issue. As opposed to bug fixing and adding new feature in my previous pull requests, this time I focused on writing the test itself, which may not have an immediate impact, but would definite improve maintainability of the project in the long run.

Finding the missing test

A comprehensive test set helps identify bugs and allows re-testing in the future to ensure that the program runs normally after updates, therefore, we should identify the missing piece in the current test set. I first went through the current test cases in the project, and spotted that there was no test case to check for the option of mix question type, which generates math questions with random operator (+,-,x,/). It would be a good idea to add a test to check if question and answer are generated properly with the mix type.

Writing the test

Now that I had decided what test to be written, I could go ahead and write the test. Similar to adding new feature, I also have to stick to original coding style when writing the test, which I have become more proficient after weeks of practice working on open source projects. A minor problem arose when I tried to push my codes after writing the test, I found that I've included the codes related to another issue that I worked on in the same repo. Then I realized that I had created a new branch from the branch for another issue instead of from the main branch. I had to re-creating a branch from the main branch, add the new test in the right branch, and eventually made a pull request.

Lesson learnt and future planning

In this pull request, I've gained experience examining existing test set, and writing test that improve the comprehensiveness of the test set. Additionally, I'm more aware of branching in git, especially when working on multiple issues on the same repo at the same time. Now I've built a habit of using git status to check the current branch first before creating new branch. Though this is my last pull request for Hacktoberfest, I will continuously look for open source projects that I'm interested in, and try my best to contribute to the open source community in the future.

Hacktoberfest 3rd Pull Request - Adding New Feature

DenisC — Thu, 30 Oct 2025 19:54:38 +0000

In the previous 2 pull requests, I have been working on bug fixing either spotted by myself or by others. This time, I have worked on adding new feature to a Math Worksheet Generator Project.

New feature added

The feature I worked on was to add a front page to the generated worksheet that contains a date/name/score section, which could be useful when the generated worksheets are distributed in class. I initially saw this in the TODO section in README.md but found no related issue in the repo, so I posted an issue regarding this feature and started working on it.

Process of adding new feature

The output worksheets are in pdf format, generated by fpdf library in Python. As I have never used this library before, I have to first read documentation on fpdf library to get the basic idea of how it works, in order to implement the new feature of adding a new page. While I was planning on how to add the name/date/score fields on the front page, I also realized that it would make sense to have a title too, so I decided to add a default title to the front page, and implement a --title option to the tool to allow customized title by user. After clearly stating my planning about the new feature in the issue, I started implementing the new feature while trying to stick to the original coding style. At last, I updated the README.md to add instruction for using the new --title option, replaced the sample-worksheet.pdf with the new one with a front page, and made a pull request.

Lesson learnt and future planning

Through working on this pull request, I have become more comfortable working with tools that I am not familiar with, such as the fpdf library this time. I realized that we don't have to fully understand everything about a tool to use it, instead, we should focus on the parts we are interested in, which would be enough for us to utilize the tools. I've also learnt the importance of having a flexible coding style, such that the codes written would be consistent to whatever projects I'm working on. So far, I've fixed bugs and added new feature in my pull requests, in my next pull request, I would like to work on a different type of issue, possibly writing a test.

Adding New Feature Inspired by Other Projects

DenisC — Wed, 29 Oct 2025 17:07:29 +0000

As mentioned in my last post about the feature I found in Repomix, I am going to add the --remove-empty-lines option to my project while making reference to Repomix. The feature is to remove blank lines from all files when writing file contents, which could be useful to reduce the file size and no. of lines in the output file, such that it does not exceed the size/line limit for AI analysis.

The first step to add this new feature is to do a prototype. As both the structure and programming language of my project is different from those of Repomix, I had to adjust the code when prototyping. Firstly, I added the new flag to the argument parser, which is a separate module in my codes. I then added a remove_empty_lines helper function to process the empty lines removal on the file content, which used the same logic as the removeEmptyLines in fileManipulate.ts of Repomix. I've also altered the write_file_content function to receive the empty_lines_removed flag as a parameter, for further processing within the function.

The changes I've made for prototyping can be tracked in this commit. What's left now is to change the logic within write_file_content function to cater for the case when the empty_lines_removed flag is true, and to update README.md after the new feature has been implemented and tested. To document the progress of implementing this feature, I have filed an issue that described the overall steps to implement the feature. After doing the basic prototyping, I've filed a sub-issue that requires changes in write_file_content function, and another sub-issue about the changes in README.md, such that other users can see the progress or help contribute to the issues.

DEV Community: DenisC

Review on My Work on Scrapy

Digging into Interrelated Issues

Working on the Issue

Reflection on My Work on Scrapy

Review of My Open Source Journey

Progress Updates on Contribution to Scrapy

About the Issue

What I Have Done

Lessons Learnt and Future Plan

Contributing to Larger Open Source Project - Scrapy

Why I chose Scrapy

My Work Plan

Expected Outcomes

Packaging my Open Source Project - Release 1.0.0

Contributing to Optimization in Open Source Project

About the issue

The progress of problem solving

Lessons learnt

Continued Journey on Open Source Projects - Deprecate Python 3.9

The issue I worked on

Challenges I faced

Lessons learnt

Setting Up GitHub Actions CI Workflow for My Open Source Project

Creating the Workflow

Challenges When Creating the Workflow

Writing Tests for Others

Lessons learnt from CI Workflow

Writing Unit Tests for My Open Source Project

Why I chose unittest framework

How I set up the tests

Difficulties writing the tests

Lessons learnt

Hacktoberfest Recap - A Month of Open Source Learning

Exploring and Getting Started

Diving Deeper into Bug Fixing

Adding a New Feature

Writing Tests

Final Reflection

Hacktoberfest Last Pull Request - Writing Test

Finding the missing test

Writing the test

Lesson learnt and future planning

Hacktoberfest 3rd Pull Request - Adding New Feature

New feature added

Process of adding new feature

Lesson learnt and future planning

Adding New Feature Inspired by Other Projects

Why I chose `unittest` framework