DEV Community: Kabir Hossain

Config precedence in Python: why your resolver shouldn't know what "CLI" means

Kabir Hossain — Wed, 14 Oct 2026 03:43:48 +0000

Most config systems I've written have the same shape. A function that looks at the environment, looks at the command line, looks at a YAML file, and decides which one wins. Something like this:

def load_config():
    if cli_args.timeout:
        return cli_args.timeout
    if os.getenv("TIMEOUT"):
        return os.getenv("TIMEOUT")
    return yaml_config.get("timeout", 30000)

Multiply that by every setting and you get a function that's 200 lines long, impossible to test in isolation, and impossible to change precedence on without editing every branch.

The version I've been working on separates two things the if/else version fuses together. What order sources apply in. And how they merge. Once those are separate, precedence stops being code and starts being structure.

The resolver doesn't know what "CLI" means. Here's the merge function in full:

def resolve_config(sources: Iterable[SourceSpec]) -> dict:
    merged_config = {}

    for spec in sources:
        try:
            config = spec.source.get_config()
        except Exception as error:
            if spec.failure_policy == FailurePolicy.REQUIRED:
                raise
            logger.warning(
                "Optional configuration source %s failed: %s",
                spec.source.__class__.__name__,
                error,
            )
            continue

        merged_config = {**merged_config, **config}

    return merged_config

Read that and notice what it doesn't do. It doesn't check whether a source is a CLI source, an env source, or a YAML source. It doesn't sort anything. It doesn't decide that one kind of source should win over another kind. It takes a list of sources, merges them left to right, and returns the result.

The last source in the list wins. That's the entire precedence rule.

That's not a limitation. It's the design. The moment the resolver knows what "CLI" means, it can start making decisions it shouldn't. It can start reordering. It can start special-casing. And once it does that, you can't test precedence without also testing the resolver's opinions, and you can't change precedence without reading the resolver's code.

Precedence is a list. That list comes from a builder:

def build_standard_pipeline(file_path: str, env: Environment, cli_args=None) -> list[SourceSpec]:
    yaml_source = YAMLSource(file_path, env)
    env_source = EnvSource()
    cli_source = CLISource(args=cli_args)
    return [
        SourceSpec(source=yaml_source, failure_policy=FailurePolicy.REQUIRED),
        SourceSpec(source=env_source, failure_policy=FailurePolicy.OPTIONAL),
        SourceSpec(source=cli_source, failure_policy=FailurePolicy.OPTIONAL),
    ]

YAML first. Env second. CLI last. Since the resolver merges left-to-right and later sources override earlier ones, CLI wins over env wins over YAML. That's the whole precedence system, and it's expressed as three lines of a list.

If you want to change precedence, say you decide env should beat CLI, you change the order of that list. Nothing else changes. No if branches to hunt down. No settings that read precedence from a config file. Just the order of the list.

The tests are where the pattern earns its keep, because they lock in behaviors, not implementation details.

First test. Later source wins.

def test_1_later_source_wins():
    source_a = FakeConfigSource({"browser": "chromium"})
    source_b = FakeConfigSource({"browser": "firefox"})
    source_c = FakeConfigSource({"browser": "webkit"})

    specs = [
        SourceSpec(source=source_a, failure_policy=FailurePolicy.REQUIRED),
        SourceSpec(source=source_b, failure_policy=FailurePolicy.REQUIRED),
        SourceSpec(source=source_c, failure_policy=FailurePolicy.REQUIRED)
    ]

    result = resolve_config(specs)
    assert result["browser"] == "webkit"

That's the base contract. Merge is left-to-right, last write wins. It's obvious in isolation and it's the thing every other behavior depends on.

Second test. Reordering changes result.

def test_2_reordering_changes_result():
    source_a = FakeConfigSource({"browser": "chromium"})
    source_b = FakeConfigSource({"browser": "firefox"})
    source_c = FakeConfigSource({"browser": "webkit"})

    specs = [
        SourceSpec(source=source_c, failure_policy=FailurePolicy.REQUIRED),
        SourceSpec(source=source_a, failure_policy=FailurePolicy.REQUIRED),
        SourceSpec(source=source_b, failure_policy=FailurePolicy.REQUIRED)
    ]

    result = resolve_config(specs)
    assert result["browser"] == "firefox"

This one looks trivial and isn't. It proves the resolver has no hidden opinion about which source should win. If someone ever helpfully adds logic to the resolver that reorders sources, say putting CLI sources last regardless of where they appear, this test fails. That's the category of mistake it prevents. Someone trying to make precedence smarter inside the merge.

Third test. The resolver doesn't sort.

def test_3_resolver_does_not_sort():
    # Simulate a logically reversed order: CLI-like -> YAML-like -> ENV-like
    # The resolver should NOT know these names mean anything.
    cli_like = FakeConfigSource({"browser": "webkit"})
    yaml_like = FakeConfigSource({"browser": "chromium"})
    env_like = FakeConfigSource({"browser": "firefox"})

    specs = [
        SourceSpec(source=cli_like, failure_policy=FailurePolicy.REQUIRED),
        SourceSpec(source=yaml_like, failure_policy=FailurePolicy.REQUIRED),
        SourceSpec(source=env_like, failure_policy=FailurePolicy.REQUIRED)
    ]

    result = resolve_config(specs)
    assert result["browser"] == "firefox"

This is the one I'd defend hardest. It uses three sources named after the standard precedence chain, CLI, YAML, ENV, but passes them in a deliberately wrong order. The resolver's job is to merge them in the order given, not to fix it.

If you ever find yourself wanting to add precedence knowledge to the resolver, this test is the one that will stop you. That's the point. Tests aren't just assertions about current behavior. They're arguments against future versions of yourself who will have worse ideas.

There's a subtle thing in the resolver that's easy to miss.

try:
    config = spec.source.get_config()
except Exception as error:
    if spec.failure_policy == FailurePolicy.REQUIRED:
        raise
    logger.warning(...)
    continue

Each source carries its own failure policy. YAML is required. If the file is missing or malformed, the pipeline fails. Env and CLI are optional. If they fail to load, the pipeline continues with what it has.

This is deliberately a separate concern from precedence. A source can be last in precedence, highest priority, and optional, failing silently. A source can be first in precedence, lowest priority, and required. The two axes are independent.

def test_5_failure_policy_remains_independent(caplog):
    valid_yaml = FakeConfigSource({"browser": "chromium", "timeout": 30000})
    failing_env = FailingSource()
    valid_cli = FakeConfigSource({"browser": "webkit"})

    specs = [
        SourceSpec(source=valid_yaml, failure_policy=FailurePolicy.REQUIRED),
        SourceSpec(source=failing_env, failure_policy=FailurePolicy.OPTIONAL),
        SourceSpec(source=valid_cli, failure_policy=FailurePolicy.REQUIRED)
    ]

    with caplog.at_level("WARNING"):
        result = resolve_config(specs)

    assert result["browser"] == "webkit"
    assert result["timeout"] == 30000
    assert "Optional configuration source" in caplog.text

Env fails. The pipeline continues. CLI still overrides YAML. Timeout from YAML still applies because nothing else set it. Precedence and failure handling do their jobs without interacting.

Fusing these two concerns is the most common mistake I see in config systems. It looks natural. If the CLI arg is missing, fall back to env, which falls back to YAML. But it collapses two orthogonal decisions into one tangled function. Keeping them separate makes each one testable in isolation and changeable without touching the other.

One more test, because it's the one that guards the whole design:

def test_7_resolver_remains_unchanged():
    sig = inspect.signature(resolve_config)
    params = list(sig.parameters.values())
    assert len(params) == 1
    assert params[0].name == "sources"

This asserts that resolve_config takes exactly one parameter. Nothing else. No pipeline name, no environment, no flags. Just the list of sources.

It sounds pedantic, and it is, deliberately. Six months from now, someone (probably me) will want to pass an Environment into the resolver so it can adjust precedence based on the environment. This test makes that change fail loudly, which forces a conversation about whether the resolver should know anything about environments. The answer is almost always no. But without the test, the answer quietly becomes yes, one small change at a time.

What this pattern buys you.

Precedence is a list. Changing it means reordering a list, not editing logic.

The resolver does one thing. It merges dictionaries left to right. That's it. It's 20 lines and it doesn't know what a CLI is.

Failure handling is orthogonal. Each source declares whether it's required or optional. The resolver respects that declaration without conflating it with priority.

Tests lock behaviors, not implementations. Each test asserts something category-level about what the system should never do, not just what it currently does.

None of this is novel. It's the same pattern behind a dozen config libraries. But the version where precedence is structure, rather than a decision buried in a conditional, is the version I've found easiest to live with. Because the day you need to change precedence is the day you least want to be reading the resolver's code.

How I scrape any website in under 5 minutes (Python + Playwright)

Kabir Hossain — Wed, 07 Oct 2026 04:48:09 +0000

python, webdev, tutorial, automation
Every scraping tutorial I've read online ends the same way.

"Here's how to grab a title with BeautifulSoup." Twenty lines of Python. Works on the demo page. Fails on a real website.

That bothered me for a long time. Because real websites are messy. They use JavaScript. They block bots. They hide data behind logins. They paginate. They rename their HTML classes every few months and break every script you've written.

So instead of writing another "20-line scraper" tutorial, I want to show you what actual production scraping looks like. Not the theory — the workflow I use for real client jobs.

What most tutorials skip
When someone asks me to scrape a site, I don't open a fresh Python file and start writing selectors. I follow a process.

Step one is an audit. I check the site first. Is it static HTML or JavaScript-rendered? Does it have clear repeating elements like product cards? Does it paginate cleanly? Does robots.txt allow scraping?

If I skip this step, I waste hours trying to scrape a site that was never going to work. Five minutes of auditing saves five hours of frustration.

Building the audit as a tool
Since I audit every site the same way, I automated the audit itself.

Here's what my tool does when I run it:

bash
tendem-audit
It asks for a URL. Then it:

Fetches the page using the same fetch strategy I use for scraping (static HTTP first, Playwright second)

Checks robots.txt

Auto-detects the repeating card block using heuristics

Fingerprints the CMS (Shopify, WooCommerce, WordPress, Magento)

Prints a verdict: green, yellow, orange, or red

Green means I can quote the job today. Red means I politely decline.

That one step — automating the audit — saved me more time than any scraper I've written.

The actual scraping
Once I know the site is scrapeable, the scraping itself is boring. That's how it should be.

bash
tendem-scrape https://shop.example.com/products --preset shopify --max-pages 50 --format csv,json,sqlite
That one command:

Fetches all 50 pages automatically by following the "Next" link

Uses the WooCommerce preset to find the right fields (title, price, image, link)

Handles the pagination URL changes

Merges items into one dataset

Writes CSV, JSON, and SQLite files

Total time: two minutes. For a 1,000-product catalog.

What production scraping actually means
The scraping part is 20% of the job. The other 80% is what makes a client pay.

Here's what I deliver for every scraping job — no exceptions:

Clean CSV, ready to open in Excel

Not a raw dump. Columns with the right names. No hidden characters. UTF-8 BOM so Excel doesn't mangle it.

Interactive HTML dashboard

A single file the client opens in their browser. Sortable columns. Filterable search box. Works offline. No dependency on my server.

Data quality report

A file that lists exactly what problems I found: missing titles, empty prices, duplicate rows, broken image links. Clients trust the data because they can see what was checked.

Audit trail

What was scraped, when, from what URL, by what method. If the client asks "how did you get this?", the answer is one JSON file away.

The technical stack

For those who want to know how it works under the hood:

Python 3.10 or newer

Playwright for anything JavaScript-rendered

BeautifulSoup with lxml for parsing

Pydantic v2 for validated data models

SQLite or PostgreSQL for storage

GitHub Actions for the CI pipeline

Pytest and Allure for tests and reports

The framework has over 70 automated tests. There's a public test dashboard on GitHub Pages that shows them passing.

What I learned doing this for real clients
Three things stand out.

First, sessions and logins matter more than anything. Half the sites I scrape require a login. If you don't learn how to save cookies and reuse them, half the job disappears.

Second, quality control is what clients actually pay for. A raw CSV is worth ten dollars. A CSV plus a quality report plus a dashboard is worth a hundred. Same data, different packaging.

Third, when a site changes, your tests catch it before the client does. That's not a nice-to-have — it's the difference between a one-off job and a monthly subscription.

Try it yourself
bash
git clone https://github.com/pranromumu/tendem-scraper
cd tendem-scraper
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
playwright install chromium

tendem-scrape https://books.toscrape.com/ --max-pages 3 --open
The last command scrapes three pages and opens the dashboard in your browser.

What's next
I'm building this out as a service for e-commerce sellers and marketing agencies who need competitor pricing, product catalogs, and lead lists on a schedule. If you have a scraping project in mind, or you just want to talk about Python automation, feel free to reach out.

The GitHub repo is open source. Fork it, break it, submit a pull request.

About me: I'm a web scraping and automation engineer based in Malaysia. I build Python tools that extract data from websites and deliver clean, structured files with quality guarantees.

Live demo: https://pranromumu.github.io/tendem-scraper/

GitHub repo: https://github.com/pranromumu/tendem-scraper

Fiverr gig: https://www.fiverr.com/s/3AAA3yL

If you found this post useful, connect with me. I write about Python, scraping, and automation for real client work.

From Script to Secure Pipeline: How I Created a "Trust Boundary" in CI/CD

Kabir Hossain — Fri, 25 Sep 2026 06:33:41 +0000

A few months ago, I thought CI/CD simply meant writing a GitHub Actions YAML file that runs pytest.
If tests pass, the result is 'green'; if they fail, it's 'red'pretty straightforward, right?
However, while progressing through my QA automation roadmap (projects P141–P170), my mentor introduced me to a concept that completely shifted my perspective on testing: the 'Trust Boundary'.
When a test fails in CI, developers need evidence (such as screenshots, traces, or logs) to debug the issue. But how can they be certain that the evidence stored hasn't been corrupted? How do they know the pipeline is making the correct decision?
Here is how I built a secure CI pipeline to address this challenge.

Preserving Evidence of Failure (The if: always()Strategy) Typically, if a GitHub Actions job fails, subsequent steps do not execute. This means that if your Playwright test crashes, the step to upload artifacts won't run-causing the failure screenshot to be lost forever as soon as the virtual machine shuts down. To solve this, I used the if: always() condition for the upload step: -name: Upload Failure Evidence uses: actions/upload-artifact@v4 if: always() with: name: test-evidence path: artifacts/ This ensures that even if the test crashes, the evidence is safely uploaded to GitHub's cloud storage.
Independent Auditor/Verifier (Decoupled from the Test Job) If the test job itself generates the evidence and also verifies it, the situation becomes akin to grading one's own homework. Even if a bug corrupts the file, the test job might still report a successful verification (verification = true) based on incorrect data. To resolve this, I split my pipeline into separate jobs: Test Job: Runs Pytest, generates evidence, calculates the SHA-256 hash, and uploads it to the cloud. Verify Job: Downloads the artifact, recalculates the hash from the raw bytes, and compares the two. This establishes a 'trust boundary.' Here, the Verify job acts as an independent auditor. If anyone (or a system bug) alters or tampers with the evidence while it is stored in the cloud, the hashes will no longer match, and the auditor will mark the pipeline as 'failed.'
The Final Quality Gate Finally, I added a third job: 'The Gate.' This job depends on the 'Verify' job (though the if: always() condition was not used). Its sole purpose is to display the final decision: ================================= FINAL EVIDENCE GATE ================================= Test execution: PASS Artifact upload: PASS SHA-256 verification: PASS FINAL GATE: PASS If verification fails, the 'Gate' step is skipped, and the overall workflow is marked as 'Red' (failed). What's next?"Praman"(Proof) By completing this stage of CI/CD, I realized that QA isn't just about finding bugs; it is about building an engineering framework based on trust and evidence. Stemming from this insight, I’ve named my future architecture'Praman'-a word that translates to 'Proof' or 'Evidence'. The next steps in my roadmap involve transforming this pipeline into a modular, "drop-in" Python QA engine that freelance clients can easily integrate into their own repositories. It will be a tool designed to execute tests, gather irrefutable evidence, and (eventually) leverage AI to analyze the root causes of failures. If you are building CI/CD pipelines, I highly recommend exploring independent artifact verification. It elevates your pipeline from a mere "script runner" into a professional-grade engineering system. What aspects of CI/CD are you currently learning? Let’s connect!

My QA Automation Journey: From Zero to CI/CD 🚀

Kabir Hossain — Mon, 21 Sep 2026 22:43:34 +0000

Hello,
I am embarking on a long journey into QA automation—and documenting my experiences along the way.
Just a few months ago, while I could run pytest, I lacked a clear understanding of what a professional automation framework actually looks like.
I was familiar with some basic concepts of Python and automation. However, there was a significant gap between knowing isolated tools and understanding how everything integrates within a real-world QA engineering workflow.
So, I decided to bridge that gap through hands-on practice.
I am following a roadmap comprising over 200 practical, hands-on projects. It begins with Python and Playwright, progressing through CI/CD, observability, and AI-assisted testing, ultimately leading to the creation of an AI-driven QA platform.
Here is what I have worked on so far:
✅ Python Fundamentals
✅ Playwright
✅ Pytest
✅ Page Object Model
✅ API Testing
✅ Authentication and Session Handling
✅ Network Interception
✅ Screenshots, Traces, Videos, and HTML Reports
✅ GitHub Actions and CI/CD

I am currently approaching Project 152 (P152).
Through this process, I have realized something important: learning QA automation isn't just about learning how to write a test.
It is equally crucial to understand what happens when a test fails. For instance, I recently set up a CI 'quality gate' for a project where the process went like this:

Tests fail
↓
CI pipeline fails
↓
Evidence or data regarding the failure is collected
↓
Artifacts are uploaded

If a test fails, the pipeline should fail too—but I didn't want the evidence or data concerning that failure to be lost along with the pipeline itself.
It was a small project, certainly, but it taught me a valuable lesson about how a real-world CI pipeline should function.
Throughout this journey, I will continue to document my project, mistakes, problem-solving approaches, and everything new I learn.