In modern Python development, modularity is king, but what happens when you know your endgame is a single, unified library? This is a situation many developers face: you're building utility packages like a logging system or metrics collector that are standalone for now, but will eventually need to be merged into a single cohesive library.
This post walks through the pros and cons of starting with a new repository vs integrating into an existing one, and why starting modular is usually the smarter long-term move.
The Setup
Let's say you're working on an internal Python project with a growing codebase. You're tasked with creating a reusable logging utility. You know this utility will:
- Be useful across multiple services
- Eventually be packaged into a broader SDK or shared library
- Possibly even released to PyPI for external use You have two choices:
- Build the logging utility inside your existing project repo
- Start fresh in a new, dedicated repository
Option 1: New Repository: The Modular Approach
Advantages:
- Cleaner package boundaries: Code is focused, self-contained, and easier to test or publish.
- Independent versioning: You can update the logging utility without waiting for the main app's release cycle.
- Professional distribution: Publishing to PyPI looks more legitimate with a clean, standalone repo.
- Better for external reuse: External developers don't have to clone or understand your entire application to use the logging package.
- Scales better over time: Easier to onboard contributors, track issues, and evolve the package independently.
Option 2: Existing Repository: The Fast Path
Advantages:
- Faster setup: No need to configure new CI/CD, permissions, or documentation.
- Easier integration: Works seamlessly with your existing code and deployment pipelines.
- Lower overhead: Everything is in one place no need to sync multiple repos.
But...
If your utility grows and you eventually need to extract it or merge it with other packages, things get messy:
- You'll face code entanglement
- Refactoring is harder once multiple modules depend on internal functions
- Clean separation for unit tests, dependencies, and CI/CD becomes challenging
My Use Case: Combining Two Wheels into One Library
In my current project, I'm building two utilities (a logging tool and another internal package). Each will be distributed as a Python wheel. Eventually, both need to be merged into a single SDK-style library.
So why not start as one package now?
Because:
- Each utility has a different dev lifecycle
- They may be used separately in the short term
- Maintaining clean separation makes future integration easier
When the time comes to combine them, I'll use:
-
Namespace packaging (e.g.,
myorg.logging,myorg.metrics) - Or a meta-package that wraps both utilities
How to Combine Later
When both packages are ready:
- Create a new umbrella repo (e.g.,
myorg-sdk) - Use
pyproject.tomlorsetup.cfgto define dependencies on both wheels - Optionally publish all under a shared namespace (
myorg.*) - Configure a single
pip install myorg-sdkexperience
This keeps the original repos intact while allowing users to install everything at once.
Takeaways
| Goal | Recommendation |
|---|---|
| External distribution | Separate repo |
| Future merging | Separate repo |
| Quick internal-only tool | Existing repo is fine |
| Long-term maintainability | Separate repo |
| Small MVP/prototype | Use existing repo to start fast |
Final Thoughts
Even if your end goal is a unified library,starting modular gives you flexibility. You'll be able to:
- Develop faster
- Maintain better code hygiene
- Evolve each component independently And when it's time to merge? You'll thank yourself for keeping things clean.
What's your approach to modular Python development? Share your experiences in the comments!
Top comments (0)