DEV Community: 137Foundry

How to Run Old and New Systems in Parallel Without Doubling Your Workload

137Foundry — Mon, 13 Jul 2026 11:34:32 +0000

Running a legacy system and its replacement side by side, comparing outputs, is the single best way to catch gaps a migration audit missed. It's also expensive if done naively, because someone has to actually watch both systems and reconcile differences. Here's how to structure a parallel run that catches real problems without burning out whoever's assigned to monitor it.

Step 1: Feed both systems the same inputs automatically

Manually re-entering the same data into two systems is slow and introduces its own errors. Set up a fork at the input layer, whether that's a message queue consumer, an API gateway, or a batch job, that sends every real input to both the legacy system and the replacement simultaneously. The legacy system's output remains authoritative during this phase; the replacement's output is being validated, not yet trusted.

def process_transaction(data):
    legacy_result = legacy_system.process(data)
    new_result = new_system.process(data)
    if legacy_result != new_result:
        log_discrepancy(data, legacy_result, new_result)
    return legacy_result  # legacy stays authoritative during validation

Photo by ThisIsEngineering on Pexels

Step 2: Automate the comparison, don't do it by eye

Diffing two outputs manually works for the first ten transactions and becomes untenable at any real volume. Write an automated comparison that flags discrepancies and logs them with enough context (the input, both outputs, a timestamp) to investigate later without needing to reproduce the exact conditions from scratch. This is the step that turns a parallel run from "hopefully someone notices something's wrong" into an actual verification process.

Step 3: Triage discrepancies by pattern, not by individual occurrence

Once discrepancies start accumulating, group them by root cause rather than investigating each one individually. A single missed edge case (a specific date range, a specific customer type) might produce hundreds of individual discrepancy log entries that are all actually the same underlying bug. Fixing the pattern once, rather than triaging each occurrence separately, is how a parallel run scales past the first few days without becoming a full-time job for whoever's assigned to it.

Step 4: Set an explicit exit criteria before you start

Decide in advance what "the parallel run is done" looks like: a specific number of consecutive days with zero unexplained discrepancies, or a specific percentage of transaction volume validated without issue. Without an explicit target, parallel runs tend to continue indefinitely out of caution, which delays the actual benefit of retiring the legacy system and keeps two systems in production maintenance simultaneously longer than necessary.

Step 5: Cover a full business cycle if the system has one

If the legacy system handles anything with a monthly, quarterly, or annual pattern (billing cycles, tax calculations, seasonal logic), make sure the parallel run's duration actually covers that cycle at least once. A parallel run that only spans two weeks will miss a bug that only manifests during month-end processing, and that gap won't surface until the legacy system is already gone and there's nothing left to compare against.

Handling discrepancies that turn out to be the legacy system's bug

Not every discrepancy means the new system is wrong. Occasionally the parallel run surfaces a case where the legacy system was actually producing an incorrect result all along, one nobody had noticed because nothing downstream ever checked it carefully. This is a genuinely useful finding, but it needs a deliberate decision, not a default assumption either way. Confirm which system is actually correct against the real business rule (not just which one is older) before deciding whether the "discrepancy" is really the new system introducing a regression or the audit accidentally surfacing a bug in a system everyone had assumed was reliable.

What this catches that a code review can't

The value of an actual parallel run, as opposed to a thorough code review of the replacement, is that it exercises real production data with all its genuine messiness: malformed inputs nobody anticipated, edge cases that occur rarely enough that no test suite happened to cover them, timing-dependent behavior that only shows up under real load. Code review catches what reviewers think to look for. A parallel run catches what actually happens.

137Foundry's engineering team builds this kind of parallel-run infrastructure as a standard phase of legacy migration work, treating it as a verification gate rather than an optional nice-to-have. The complete legacy retirement framework covers how this fits alongside the audit and knowledge-recovery work that should happen before the parallel run even starts.

References

Wikipedia's entry on legacy systems for background on why these systems accumulate the kind of edge-case behavior that a parallel run is designed to catch.
Martin Fowler's website covers software architecture patterns including strangler fig migrations, a closely related approach to gradually replacing a legacy system rather than a single cutover.
MDN's web docs is a useful general reference if the comparison layer you're building involves web APIs or HTTP-level request forking.

A parallel run is real infrastructure work, not a formality. Budgeted properly, it's also the cheapest insurance available against shipping a replacement system that's subtly, silently wrong in ways nobody catches until months later.

Deciding when a parallel run isn't worth building

Not every migration justifies this level of infrastructure. For a low-volume, low-stakes internal system with a thorough audit behind it, the engineering cost of building input forking and automated comparison can exceed the actual risk being mitigated. Reserve the full parallel-run treatment for systems where a wrong answer has real business consequences, financial, legal, or customer-facing, and lean on a simpler validation approach, like spot-checking a sample of transactions manually, for lower-stakes systems where the infrastructure investment wouldn't pay for itself.

A note on team morale during the validation period

Running two systems in parallel for weeks or months is genuinely tedious for whoever's assigned to monitor it, especially once the interesting discrepancies have mostly been found and fixed and what remains is mostly confirming that things continue to work correctly. Rotating this responsibility across the team, rather than assigning it permanently to one person, keeps the monitoring rigorous instead of becoming an afterthought that one burned-out engineer checks less and less carefully as the weeks go on. It's a small operational detail, but it's one of the more common reasons parallel runs quietly stop catching real problems well before their planned end date.

Pair the rotation with a lightweight weekly summary, even a few sentences, of what the past week's discrepancies looked like and whether the rate is trending down. This keeps the whole team oriented on whether the validation period is actually converging toward "safe to cut over" or stalling on a recurring issue that needs deeper investigation, rather than everyone individually assuming someone else is tracking the trend.

A short summary is enough. The point isn't a formal report, it's making sure the trend is visible to the whole team rather than living only in whoever happens to be on monitoring duty that particular week, which is what actually lets the team collectively decide when the exit criteria has genuinely been met, rather than one person's gut feeling carrying that decision alone.

7 Free Tools for Auditing an Old Codebase Before You Migrate It

137Foundry — Mon, 13 Jul 2026 11:34:30 +0000

Before replacing a legacy system, you need an honest picture of what it actually does, not what the original documentation says it does or what everyone assumes. These free tools cover the inspection work that should happen before a single line of replacement code gets written.

Photo by Brett Sayles on Pexels

1. Your version control's blame and log commands

git log --follow and git blame on the files that make up the core business logic tell you when a piece of code was introduced and, often, why, if the original commit message was written with any care. This is free, it's already in your repository, and it's the fastest way to distinguish a rule that's been stable for a decade from one that was a quick patch six weeks ago. Git's own documentation covers both commands if your team isn't already using them for this kind of archaeology.

2. A static analysis tool for dead code detection

Legacy systems accumulate code paths that no longer execute, conditionals guarding for a state that can't happen anymore, whole functions nothing calls. Tools like Vulture for Python or built-in dead code warnings in most modern compilers help separate logic that's still load-bearing from logic that's safe to simply not carry forward into a replacement.

3. A dependency graph visualizer

Understanding what actually calls what, especially in a codebase that's grown organically over many years, is much easier with a visual dependency graph than by reading files in whatever order they happen to be organized. Tools like Graphviz can render a call graph or module dependency graph from most languages' static analysis output, making it far easier to spot which parts of the system are actually central versus which are isolated, rarely touched, and lower-risk to leave for last.

4. A test coverage report, even a rough one

Legacy systems are notorious for having little or no automated test coverage, but generating even a rough coverage report (which lines execute during a typical run, which never do) tells you where the system's actual behavior is exercised regularly versus where it's dormant. Dormant code paths deserve extra scrutiny during a migration precisely because nobody's recently confirmed they still work correctly.

5. A database schema visualizer

If the legacy system's business logic lives partly in stored procedures, triggers, or an unusually complex schema (common in systems old enough to predate modern ORMs), a schema visualization tool like SchemaSpy surfaces relationships and constraints that aren't obvious from reading application code alone. This matters especially for systems where business rules were implemented at the database layer rather than the application layer, which is a common pattern in systems built before that separation became standard practice.

6. A log aggregator, even a temporary one

If the legacy system produces logs at all, pulling them into a searchable aggregator, even a temporary open-source one like the ELK stack (Elastic's website) or a simpler alternative, lets you search for patterns across months of production history instead of reading log files one at a time. This is especially useful for confirming how often a specific rare code path actually fires, information that directly feeds into the frequency-times-risk prioritization that should guide how deep an audit goes on any individual finding.

7. A documentation generator for whatever comments do exist

Even sparse or outdated inline comments are worth extracting into a single searchable document rather than left scattered across hundreds of files. Tools like Doxygen or language-specific documentation generators pull every comment into one browsable reference, which makes it much faster to spot patterns (a comment style that shows up around every workaround, for instance) than reading file by file.

Putting these together with the human side of the audit

None of these tools replace talking to the people who currently use or maintain the system. What they do is make those conversations more productive, because you can ask specific, informed questions ("I see this function hasn't been touched in eight years and nothing in our test suite exercises it, what does it actually do") instead of vague ones. The tooling surfaces where to look. The interviews explain why what you're looking at exists in the first place.

For a fuller framework on how this technical audit fits into a complete legacy retirement process, including how to prioritize what to preserve and how to structure a parallel-run verification period before fully decommissioning the old system, see https://137foundry.com's guide on sunsetting a legacy system without losing institutional knowledge.

A note on scope

None of these five tools are expensive or complicated to set up, which is part of the point. The barrier to a proper legacy audit is rarely tooling. It's making the time for it in a project plan that's usually under pressure to get to the "real" migration work as quickly as possible. Running these tools takes an afternoon. Skipping them and discovering the gaps in production takes considerably longer, spread out over months of confusing, hard-to-diagnose incidents.

Budget the audit phase as its own line item, with its own estimated hours, rather than folding it silently into "planning," where it tends to get compressed first whenever a deadline gets tight.

Combining tool output with human context

None of these seven tools tell you why a piece of logic exists, only that it exists and roughly how often it runs. The tooling narrows down where the interesting questions are; answering them still requires talking to whoever's closest to the system. A useful workflow is running the technical audit first, compiling a short list of the specific business-logic questions it raised, and then bringing that concrete list into interviews rather than starting the interviews cold. People give much more specific, useful answers when asked "why does this specific condition on line 340 exist" than when asked a general "tell me everything you know about this system."

Keeping the audit from becoming its own multi-month project

There's a real risk of over-investing in tooling and analysis to the point where the audit itself becomes the bottleneck. Time-box each tool's pass: a day or two for the dependency graph, a day for the dead code scan, rather than letting any single step expand indefinitely. The goal is a good-enough map of the system's actual behavior to inform the interviews and the migration plan, not an exhaustive, perfectly complete catalog of every line of code.

A reasonable target for a mid-sized legacy system is a week of tooling-based audit work before moving into the interview phase. If a specific tool's pass is taking noticeably longer than that without converging on useful findings, that's usually a signal to move on with what you have rather than keep digging for diminishing returns.

Keep a running list of open questions the tooling couldn't answer on its own. That list becomes the actual agenda for the interview phase, which keeps those conversations focused and efficient instead of open-ended and hard to schedule around, and it respects the interviewee's time by showing up with specific questions rather than an open-ended request to "tell us everything."

How to Write a CLAUDE.md or .cursorrules File That Actually Gets Followed

137Foundry — Sun, 12 Jul 2026 11:30:48 +0000

Plenty of teams have a CLAUDE.md or .cursorrules file sitting in their repo root that nobody has touched since the day they created it, and it shows in the output. Writing the file isn't the hard part. Writing one specific enough to change behavior, without becoming so long it gets skimmed, is the actual skill.

Step 1: Audit What You're Actually Repeating in Chat

Before writing anything, spend a week paying attention to what you type into the assistant chat over and over. "Use our ApiError class, not bare exceptions." "We're on Vitest, not Jest." "Don't touch the billing module without asking." Whatever you find yourself typing more than twice belongs in the file. Everything else is probably noise.

This audit step matters because it grounds the file in real friction instead of a generic checklist copied from someone else's repo. A file built from your team's actual repeated corrections is shorter and more effective than one built from guessing what might be useful.

Step 2: Separate Durable Rules From Temporary Ones

"We use PostgreSQL" is a durable rule. "We're mid-migration off the old billing service, don't extend it" is temporary and needs a review date. Mixing the two without marking which is which means the temporary rules quietly outlive their relevance and start confusing new team members and assistant sessions alike, sometimes for years after the migration finished.

A simple convention works well here: put temporary rules in their own section with a target removal date in a comment. Review that section on a schedule, even a rough quarterly one, and delete what's no longer true.

Step 3: Point at Examples Instead of Describing Patterns Abstractly

"Follow the pattern in handlers/users.py for new API handlers" is more effective than three paragraphs describing what a good handler looks like. Assistants read code better than they read abstract prose about code. If your codebase already has a clean example of the pattern you want repeated, cite the file path directly.

This only works if the example file is actually current. An outdated example is worse than no example, since it actively points the assistant toward a pattern you've since moved away from.

Step 4: Add an Explicit "Ask First" List

Name the modules, files, or operations that need a human in the loop before the assistant touches them: schema migrations, auth logic, anything touching payment processing. This is the single highest-value line item for teams worried about AI-generated changes causing real damage, and it's often missing entirely from instructions files that otherwise look thorough.

This list matters more than almost anything else in the file precisely because the failure mode it prevents is asymmetric. A wrong guess about which HTTP client to use costs a few minutes in review. A wrong guess about a database migration or an authorization check can cost considerably more, and an assistant with no signal that a file is dangerous treats it with exactly the same confidence it applies to a throwaway script.

Step 4b: Test the Rules Against a Real Task

Before considering the file finished, run one real task through it and check whether the assistant actually followed the rules you wrote. This sounds obvious, but it's the step most teams skip, and it's the fastest way to find out that a rule you thought was clear was actually ambiguous. If the assistant still reaches for the wrong library after you've told it not to, the instruction probably needs a concrete file reference rather than a general statement, since assistants tend to follow specific pointers to real code more reliably than abstract descriptions of intent.

Step 5: Keep It Under a Page and Revisit It in Pull Requests

A 2,000-word instructions file gets skimmed by humans and effectively deprioritized by the assistant's limited attention to any single piece of context. Aim for something a new hire could read in two minutes. When a convention changes, update the file in the same pull request that changes the code, not as a separate cleanup task that never happens.

"The instructions files that actually work are the boring ones, five or six specific rules instead of an essay about engineering philosophy." - Dennis Traina, founder of 137Foundry

Step 6: Treat It Like Any Other Documentation Debt

Instructions files rot the same way READMEs rot: quietly, until someone notices the assistant is confidently doing something the team stopped doing months ago. Assign it an owner, even an informal one, and check it during any sprint where the team makes a significant architectural decision.

Version control history is your friend here. If you're not sure whether a rule is still accurate, git log on the file itself usually tells you when it was last touched and by whom, which is a fast way to spot the stale sections before they cause a confusing pull request.

Where This Fits With the Rest of Your Tooling

An instructions file isn't a replacement for the mechanical tooling your team should already have in place. ESLint and Prettier handle formatting and a class of common mistakes automatically, which frees the instructions file to focus on the judgment calls a linter can't make: which of two valid architectural patterns your team has actually settled on, and which parts of the codebase need extra caution.

Whichever tool your team standardizes on, whether that's Cursor or something else entirely, the instructions file format has converged enough across the ecosystem that the same document usually works with minor adjustments regardless of which assistant reads it. That portability is one more reason the file is worth getting right the first time instead of treating it as tool-specific throwaway configuration.

A Realistic Timeline

None of this needs to happen in one sitting. A reasonable rollout looks like a rough first draft in an afternoon, based on whatever repeated corrections you can recall off the top of your head, followed by two or three weeks of quietly noting new corrections as they come up in real sessions. At that point, a short revision pass usually captures the rules that actually matter, and the file is in good enough shape to stop actively iterating on and start just maintaining.

Trying to get it perfect on the first attempt tends to backfire, producing either an overlong document nobody reads or a set of rules based on guesses about what might matter rather than what actually does. The iterative version, grounded in real repeated friction, consistently ends up shorter, more specific, and more likely to actually change the assistant's output.

When to Consider It Done

There's no permanent finished state for a file like this, but there is a reasonable point to stop actively iterating and shift to maintenance mode: when new corrections in chat sessions become rare rather than routine. At that point the file has captured most of what the team actually needed documented, and further additions are more likely to be edge cases that don't generalize well than genuinely useful rules. Keep the maintenance habit, the update-it-in-the-same-pull-request discipline, but stop treating the file as a work in progress once it's earned that stability.

We cover the broader codebase structure that makes an instructions file effective, not just present, in our guide to structuring a codebase so AI coding assistants stop guessing.

The development team at 137Foundry has written and maintained these files across a range of client codebases, and the pattern that works is consistently the same: short, specific, and updated in the same breath as the code it describes.

7 Free Tools for Auditing What Your AI Coding Assistant Actually Changed

137Foundry — Sun, 12 Jul 2026 11:30:44 +0000

Trusting an AI coding assistant's summary of its own changes is a mistake most teams only make once. The summary says "updated the validation logic," the diff says it also touched three unrelated files and quietly changed a default timeout. These tools make the actual diff impossible to miss.

1. Git Itself

The most underused audit tool is the one already installed. git diff --stat before you even open a pull request tells you the blast radius of a change in one line. If an assistant claims a "small fix" and the stat shows twelve files touched, that's worth a second look before anything else.

2. GitHub Pull Request Review Tools

GitHub's native diff view, combined with required reviewers on any AI-assisted branch, is still the backbone of most teams' audit process. Pair it with branch protection rules so nothing merges without at least one human set of eyes, regardless of how confident the assistant's own commit message sounds.

3. ESLint

Configured with rules against unused variables, implicit any types, and suspicious patterns, ESLint catches the class of mistake assistants make when they're pattern-matching quickly: leftover debug code, unused imports from an abandoned approach, or subtly wrong comparison operators.

4. Prettier

Not a bug catcher on its own, but a fast way to normalize formatting differences so a human reviewer's diff isn't cluttered with whitespace noise next to the actual logic change. Run it as a pre-commit step so every diff, human or AI-assisted, arrives in the same shape.

5. Static Analysis via SonarQube or Similar

A static analysis pass flags the categories of mistakes both a rushed human reviewer and the assistant itself are prone to miss: security-sensitive patterns, unreachable code, and complexity spikes in functions that used to be simple before a "quick fix" ballooned them.

6. Vitest or Your Existing Test Runner, Run on Every AI-Assisted Commit

Tests are the fastest audit tool you already have if your suite has real coverage. Running the full suite, not just the tests touching the changed files, catches the cases where an assistant's change had a side effect nobody anticipated in a file three layers removed from the one it edited.

7. A Simple Changelog Convention

Not a tool exactly, but worth including here because it changes how every other tool on this list gets used: require a one-line note in the pull request description explaining why the change was made, written by the human who reviewed it, not copied from the assistant's own summary. Six months later, this is the difference between understanding a change at a glance and reverse-engineering the reasoning from the diff alone.

Putting These Together Instead of Picking One

None of these tools replace each other, and none of them are sufficient on their own. Git's diff stat tells you the blast radius but not the quality. ESLint catches mechanical mistakes but has no idea whether the logic is right for your specific business rules. Tests catch behavioral regressions but only for the paths someone thought to cover. The real audit process is all seven working together, each catching a different category of problem the others were never designed to catch.

A useful way to think about ordering: run the fast, cheap checks first. Prettier and ESLint take seconds and catch a meaningful share of issues before a human reviewer even opens the diff. Save the expensive step, a careful human read of the full changeset, for last, after the mechanical checks have already cleared out the noise.

How Often Teams Skip This and Regret It

The most common failure pattern isn't skipping all seven tools. It's skipping the review step specifically because the assistant's own summary sounded confident and complete. A summary that says "small refactor, no behavior change" is exactly the kind of claim that deserves the full diff read, not less scrutiny, because a confidently wrong summary is more dangerous than an honest "not sure, please check this carefully."

Teams that build the habit of reading the actual diff before trusting any summary, human-written or assistant-written, catch a disproportionate share of the subtle bugs that would otherwise ship. It costs a few extra minutes per pull request and saves considerably more than that the first time it catches something real.

Making the Habit Stick Past the First Month

Most teams adopt some version of this checklist enthusiastically for a few weeks after a close call, then quietly let it slide once the memory of the incident fades. The tools that survive long-term are usually the ones wired into CI so they run automatically rather than the ones that depend on an individual remembering to run them manually before every merge. If ESLint, Prettier, and the test suite all run on every pull request without anyone having to trigger them by hand, the habit persists long after the original motivating incident is forgotten.

The human steps, reading the full diff and writing an honest changelog line, are harder to automate and easier to let slip. Making them a required part of the pull request template, rather than a suggested best practice, is a small structural change that keeps them from quietly disappearing the way purely voluntary habits tend to.

What This Looks Like Applied Consistently

A team that runs all seven checks on every AI-assisted pull request, without exceptions for changes that look small, ends up with a noticeably different failure profile than a team that applies these checks selectively based on how risky a change seems at a glance. The selective approach fails specifically because risk is hard to judge accurately from a summary alone, and the changes that turn out to matter most are rarely the ones that looked dangerous going in.

Consistency also has a second-order benefit: it makes the review process predictable for everyone on the team, including new hires who haven't yet developed the instinct for which changes deserve extra scrutiny. A fixed checklist applied every time removes the need for that instinct to exist at all, which is a more reliable foundation than hoping everyone's judgment calibrates the same way.

Not a tool exactly, but worth including: require a one-line note in the pull request description explaining why the change was made, written by the human who reviewed it, not copied from the assistant's own summary. Six months later, this is the difference between understanding a change and reverse-engineering it from the diff alone.

"The audit habit that actually sticks is the boring one: read the full diff before you read the summary, every single time, no exceptions for small changes." - Dennis Traina, founder of 137Foundry

None of these tools require the assistant to be trustworthy on its own. That's the point. A codebase with good structure, documented conventions, and a real audit habit gets more reliable output from AI coding assistants precisely because nobody is relying on the assistant to police itself.

We cover the codebase structure side of this, the files and conventions that reduce how often you need to reach for these audit tools in the first place, in our guide to structuring a codebase so AI coding assistants stop guessing.

137Foundry helps engineering teams set up both sides of this: the conventions that prevent bad output and the review process that catches what slips through anyway.

What Actually Happens When You Paginate a Live, Growing Table

137Foundry — Sat, 11 Jul 2026 11:29:55 +0000

Pagination bugs are some of the hardest to catch in code review, because the code looks correct. It passes tests. It works perfectly in staging, where the data is static and nobody is writing to the table while QA clicks through pages. Then it ships, and somewhere in production a support agent notices a customer record that keeps disappearing from page 2 and reappearing on page 3 an hour later. Nobody touched the record. The table just kept growing while someone was scrolling through it.

The mental model that causes the bug

Most developers think of pagination as slicing a fixed list: rows 1 through 25, then 26 through 50, and so on, the same way you'd paginate a static array in memory. That model is accurate for an array that isn't changing. It's inaccurate for a database table under active writes, because the "list" isn't fixed at all. New rows are being inserted, and sometimes deleted, between the moment a client requests page 1 and the moment it requests page 2, and the pagination code has no idea any of that happened.

A concrete walkthrough

Say a table is sorted by created_at DESC, newest first, and a client is paging through it with OFFSET/LIMIT. They fetch page 1: rows 1 to 25, the 25 most recent records at that moment. Before they request page 2, three new rows get inserted somewhere else in the system, a normal amount of write traffic for any active application. Those three new rows are now the newest records, which pushes everything else down by three positions in the sort order.

When the client requests page 2 with OFFSET 25 LIMIT 25, the database doesn't know or care that three new rows showed up in the meantime. It just skips the first 25 rows of the current result set and returns the next 25. But the current result set has shifted underneath the client. The three rows that used to sit at positions 23, 24, and 25 (visible on page 1) are now at positions 26, 27, and 28, which means they get skipped entirely and never shown to the client on either page. Three real records silently vanish from the user's view, and there's no error, no warning, nothing in the logs to flag it. The pagination "worked" exactly as designed. It just wasn't designed for a moving target, and that distinction matters enormously once real traffic hits it.

Why this is worse than it sounds

This isn't a rare edge case that only matters at extreme scale. Any table with regular writes, orders, support tickets, activity logs, comments, notifications, is affected the moment two users are browsing the same paginated list at the same time as writes are happening anywhere in that table. On a high-traffic application, that's not an edge case at all. That's every page load, every single day, quietly producing slightly wrong results that almost nobody notices until a customer complains about a specific missing record.

The fix: stop asking "skip N," start asking "after this point"

Cursor-based (also called keyset) pagination replaces the offset with a pointer to a specific row, built from the sort column plus a unique tiebreaker like the primary key:

SELECT * FROM orders
WHERE (created_at, id) < (:last_created_at, :last_id)
ORDER BY created_at DESC, id DESC
LIMIT 25;

This query doesn't care how many rows exist before the cursor position. It doesn't skip anything at all. It seeks directly to the row identified by the cursor and reads forward. New inserts elsewhere in the table don't shift this query's results, because the query isn't defined in terms of position, it's defined in terms of a specific row's sort key, which stays valid no matter how many other rows get added or removed around it. The PostgreSQL project's official site documents how this seek pattern uses a composite index far more efficiently than an offset-based scan, if you want to verify the mechanism yourself with EXPLAIN ANALYZE against your own schema.

The tiebreaker is not optional

A cursor built from created_at alone breaks the instant two rows share a timestamp, which happens constantly at millisecond precision under real concurrency, especially on bulk-import or batch-write workloads where many rows land in the same transaction. Without a unique tiebreaker, ties get resolved arbitrarily by the database's internal storage order, which reintroduces the exact skip-or-duplicate bug cursor pagination is supposed to fix in the first place. Always pair the sort column with a unique column, almost always the primary key, so the ordering is strictly deterministic regardless of how many rows share the same timestamp.

Testing for this specific failure mode

Static test fixtures won't catch this, because the bug only exists under concurrent writes. Write a test that inserts new rows between page requests and asserts every row from page 1 and page 2 combined is unique, no repeats, no gaps compared to what was actually in the table. That's the one test that actually exercises the bug this whole class of pagination problem produces, and it's cheap to write once you know what you're testing for. Tools like pytest make this straightforward to set up as a repeatable integration test against a real database instance rather than a mock, and for JavaScript backends, Jest covers the same kind of setup against a real test database.

A variant of the same bug: deletions instead of insertions

Everything above focuses on insertions shifting the result set, but deletions cause a mirror-image version of the same problem. If a row gets deleted from earlier in the sort order while a client is mid-pagination, offset pagination will skip one row too many on the next request, since the position it's counting from has shifted backward instead of forward. The client silently misses whatever row would have been at that boundary. This is common on tables where rows get soft-deleted or archived out of the default query scope while users are actively browsing, like a task list where completed items get filtered out as they're marked done in real time by other users.

Cursor pagination handles this case the same way it handles insertions: because the cursor identifies a specific row's sort key rather than a numeric position, a deletion elsewhere in the table simply doesn't affect where the next query resumes from.

The broader lesson

This bug is a good example of why "it passed all the tests" and "it's correct" are not the same claim. The tests that existed were testing the wrong thing: static correctness on a snapshot, not correctness under the conditions the code will actually run in. Any pagination implementation destined for a table with regular writes needs a test that simulates those writes happening mid-pagination, or the bug simply won't surface until a real user hits it in production.

We cover the full implementation, including cursor encoding, backward pagination, and the index you need to make it fast, in a longer guide over on 137foundry.com. Worth a read before your next list endpoint ships against a table that's going to keep growing.

A checklist before you ship any paginated endpoint

Before merging a new list endpoint, it's worth running through a short checklist rather than trusting that "it returned the right rows in my manual test" is sufficient proof of correctness. Confirm the sort clause includes a unique tiebreaker alongside the primary sort column, not just the primary sort column alone. Confirm there's a composite index backing that exact sort order, verified with an actual query plan rather than assumed from the schema. Confirm a test exists that inserts or deletes rows mid-pagination and checks for duplicates or gaps across page boundaries. And confirm the API documentation, whether internal or public, states plainly whether the endpoint uses offset or cursor pagination, since client code that assumes the wrong one will misbehave in ways that are hard to trace back to the actual cause.

None of these four checks take more than a few minutes once you know to look for them, and together they cover almost every version of this bug we've seen show up in real applications. The alternative, discovering the problem after a customer reports vanishing data, costs a lot more than a few minutes, both in engineering time spent tracing the cause and in the trust cost of a customer wondering what else might be wrong with a system they're relying on. Building the check into a pull request template or a pre-merge checklist is a cheap way to make sure it doesn't depend on any one engineer remembering to think about it every time on every new endpoint.

How to Build API Pagination That Survives Rate Limits and Client Retries

137Foundry — Sat, 11 Jul 2026 11:28:04 +0000

A pagination implementation that works fine in a demo can fall apart the moment a real client starts retrying failed requests, backing off, and resuming where it left off. If your cursor or page token isn't resilient to retries, you end up with duplicate processing, skipped records, or client-side bugs that only show up under real production network conditions, usually discovered by whichever customer has the flakiest network connection. Here's how to build it so it holds up under exactly those conditions.

Step 1: Make the cursor idempotent, not session-based

The most common mistake is tying a pagination cursor to server-side session state, like an in-memory query result cached against a session ID or a temporary server-side result set. The moment a client retries after a timeout, or a load balancer routes the retry to a different server instance than the one that generated the cursor, that session state is gone and the request fails or returns garbage instead of the expected next page.

Instead, make the cursor fully self-describing. Encode everything needed to resume the query directly in the cursor itself: the sort column value and a unique tiebreaker, typically the primary key.

import base64
import json

def encode_cursor(sort_value: str, row_id: int) -> str:
    payload = json.dumps({"sort_value": sort_value, "id": row_id})
    return base64.urlsafe_b64encode(payload.encode()).decode()

Any server instance, at any time, can decode this cursor and resume the exact same query from the exact same point. No session state required anywhere in the stack, which also makes horizontal scaling of your API servers trivial since no instance needs to remember anything about a client's previous requests.

Step 2: Return rate limit headers on every paginated response

If your API enforces rate limits, and most public APIs handling meaningful traffic should, include standard headers so clients can back off intelligently instead of guessing when it's safe to retry:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 43
X-RateLimit-Reset: 1720713600

There's no single universal standard for these exact header names, but this pattern is common enough across major APIs that most client libraries already know how to read it out of the box. Document your exact header names clearly in your API reference, since a client that can't parse your rate limit headers will retry blindly on a fixed interval and make the underlying problem worse rather than better.

Step 3: Make retries safe by design, not by convention

A client that times out waiting for page 3 and retries with the exact same cursor should get the exact same page 3 back, not a different set of rows. Because cursor-based pagination is deterministic (same cursor, same query, same result, assuming no underlying data changed in the meantime), this is naturally idempotent as long as you're not accidentally injecting randomness like ORDER BY RANDOM() or a non-deterministic tiebreaker somewhere in the query. Double-check your sort clause always includes a unique column so ties resolve the same way on every single execution.

Step 4: Implement exponential backoff on the client side

If you're consuming a paginated API rather than building one, implement backoff with jitter rather than fixed-interval retries, especially when pulling large datasets across many sequential pages:

import time
import random

def fetch_with_backoff(fetch_fn, max_retries=5):
    for attempt in range(max_retries):
        try:
            return fetch_fn()
        except RateLimitError:
            wait = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)
    raise Exception("Max retries exceeded")

Jitter matters because without it, many clients hitting a rate limit at the same moment will all retry in lockstep, creating a thundering herd effect that makes the rate limit problem measurably worse instead of resolving it. The Python requests library documentation covers session-level retry configuration if you'd rather not hand-roll this logic yourself.

Step 5: Set a reasonable page size ceiling to reduce total request volume

Smaller default page sizes mean more total requests needed to page through a large dataset, which means more chances to hit a rate limit somewhere in the middle of a long pull. If you know a client is doing a bulk export or a full data sync, a larger page size, within your API's hard maximum, reduces both the number of round trips and the number of opportunities for a transient failure to interrupt the process partway through.

Step 6: Log cursor state on failure for debugging and safe resumption

When a paginated pull fails partway through, log the last successful cursor so the operation can resume from that exact point rather than restarting from page 1. This matters enormously for large data syncs where restarting from scratch could mean re-processing millions of already-handled rows, which is both slow and risks duplicate side effects if the downstream processing isn't itself idempotent.

Step 7: Version the cursor format so future changes don't break old clients

Once your API has real clients holding onto cursors, whether that's a bookmarked "load more" position or a long-running batch job resuming after a failure, changing the cursor's internal shape can silently break anyone with an old cursor still in hand. Include a version marker in the encoded payload so your decoder can detect an outdated format and respond with a clear, actionable error rather than a confusing failure deep in a query:

def decode_cursor(cursor: str) -> dict:
    payload = json.loads(base64.urlsafe_b64decode(cursor.encode()))
    if payload.get("v") != 1:
        raise ValueError("unsupported cursor version, please request a fresh page")
    return payload

This is a small amount of upfront work that saves a confusing debugging session later, whenever the schema underneath the cursor inevitably needs to change to support a new sort option or filter combination.

Step 8: Handle partial failures in bulk consumers gracefully

If a client is using pagination to pull a large dataset for a batch job, sync, or export, design the consumer so a failure on page 400 doesn't discard the 399 pages already successfully processed. This sounds obvious written out, but it's a surprisingly common failure mode in hastily written sync scripts that wrap the entire pagination loop in a single try/except block and throw away all progress on any exception, including transient ones that a retry would have resolved cleanly.

def sync_all_pages(fetch_fn):
    cursor = None
    processed = 0
    while True:
        try:
            page = fetch_with_backoff(lambda: fetch_fn(cursor))
        except Exception as e:
            log_failure(last_cursor=cursor, error=e, processed_so_far=processed)
            raise
        process_records(page["data"])
        processed += len(page["data"])
        if not page["has_more"]:
            break
        cursor = page["next_cursor"]
    return processed

The key detail is logging the last successful cursor before re-raising, which ties directly back into step 6 and gives whoever restarts the job a clean resumption point instead of a guess.

Putting it together

None of these six steps depend heavily on each other, so they can be implemented incrementally and tested one at a time. The highest-leverage change, by far, is step 1: making the cursor self-describing rather than session-dependent. Everything else builds resilience on top of that foundation, and none of it matters much if the cursor itself can't survive a request landing on a different server instance. If your API's pagination breaks under retries today, that's almost always where the root cause actually lives, not in the retry logic itself.

For teams building or auditing API infrastructure like this, 137Foundry's web development service covers exactly this kind of backend design work, from initial architecture through production hardening and load testing.

Seven Free Tools for Sandboxing What an AI Coding Assistant Can Execute

137Foundry — Fri, 10 Jul 2026 09:27:11 +0000

Giving an AI coding assistant somewhere safe to actually run things, not just read and suggest, is one of the better investments a team can make. A disposable environment turns "trust the assistant to be careful" into "it literally cannot reach anything that matters," which is a much sturdier guarantee. These seven tools cover most of what a small team needs to build that sandbox without a lot of infrastructure overhead.

1. Docker

Docker is the baseline for isolating what a process, human or assistant driven, can reach. Run the assistant's shell inside a container with no network route to production and no mounted credentials, and an entire category of "wrong environment" incidents becomes structurally impossible rather than merely discouraged.

2. Docker Compose

Docker Compose makes disposable multi-service environments trivial to spin up and tear down. A scratch database, a scratch cache, and a scratch app server, seeded with synthetic fixtures, gives an assistant a real environment to test migrations against without any real data anywhere nearby.

3. Testcontainers

Testcontainers spins up throwaway, real instances of databases and services specifically for test runs, then tears them down automatically. It's a clean fit for letting an assistant apply and verify a migration against an actual Postgres or MySQL instance that exists for exactly as long as the test does.

4. SQLite

For lighter-weight local iteration, SQLite needs no server process at all, just a file, which makes it an easy default for an assistant's early draft-and-iterate loop before a migration gets tested against the real database engine your production stack actually uses.

5. act

act runs your GitHub Actions workflows locally, which means an assistant, or a human, can validate a CI change without pushing to a branch and waiting on a real pipeline run. Useful for keeping assistant-proposed CI changes in a fast, disposable loop before they touch the shared pipeline.

6. Firecracker

Firecracker is the microVM technology behind several serverless sandboxing platforms, worth knowing about if you're building a more serious isolated execution environment for assistant-triggered code rather than relying on container isolation alone. It's a heavier lift to adopt directly, but it's the technology underneath a lot of the managed sandboxes teams reach for instead.

7. Faker libraries (Faker.js / Python Faker)

Synthetic test data matters as much as environment isolation. Faker and its Python equivalent generate realistic fake data so an assistant's sandbox has something meaningful to run against, without ever touching a copy of real customer records, which closes off the specific failure mode where "isolated" environments quietly get seeded from a production dump.

Choosing between these based on team size

A two or three person team doesn't need Firecracker or a serious microVM setup, that's solving a problem at a scale you don't have yet. Docker Compose plus SQLite covers most early-stage sandboxing needs with almost no setup overhead, and you can add Testcontainers once your test suite is actually asserting against real database behavior rather than an in-memory approximation. Save the heavier infrastructure, Firecracker specifically, for when you're running many concurrent assistant sessions and need strong isolation guarantees between them, which is a later-stage problem for most teams, not a starting one.

What a minimal setup actually looks like end to end

A reasonable starting configuration for a small team: a Compose file defining a scratch Postgres instance and the app itself, a seed script using Faker to populate that instance with realistic but synthetic data, and an assistant configuration that points every database-touching command at the Compose instance's connection string, never the real one. That's roughly a day of setup work for most stacks, and it closes off the majority of "assistant ran something against the wrong database" incidents structurally, rather than relying on the assistant, or a human, remembering to double check the connection string every time.

# docker-compose.yml, minimal sandboxing setup
services:
  scratch_db:
    image: postgres:16
    environment:
      POSTGRES_DB: scratch
      POSTGRES_PASSWORD: local_only
    ports:
      - "5433:5432"

# seed.py, using Faker to populate the scratch database with synthetic data
from faker import Faker
import psycopg2

fake = Faker()
conn = psycopg2.connect("postgresql://postgres:local_only@localhost:5433/scratch")
cur = conn.cursor()

for _ in range(200):
    cur.execute(
        "INSERT INTO customers (name, email) VALUES (%s, %s)",
        (fake.name(), fake.email()),
    )
conn.commit()

Point every assistant-triggered database command at localhost:5433 during drafting and testing, and the question of whether it's safe to let the assistant run a migration stops being a trust question and becomes a fact about which connection string is configured, which is a much easier thing to get right consistently.

Extending the setup as the project grows

The minimal setup above is a reasonable starting point, but it's worth planning for what changes as a project matures. A single scratch Postgres instance works fine for one assistant session at a time. Once multiple contributors, human or assistant-driven, are working in parallel, each needs its own isolated instance rather than sharing one scratch database where one session's test data pollutes another's results. Docker Compose profiles or a lightweight per-branch provisioning script handle this reasonably well without needing to jump straight to something like Firecracker.

It's also worth thinking about how the synthetic seed data evolves alongside the real schema. A seed script written once at project start will drift from reality as new tables and columns get added over time. Treat updates to the seed script as part of the normal migration review process, not a separate maintenance task, so the sandbox stays representative of what the assistant will actually encounter when it drafts a new migration against the current schema rather than an outdated approximation of it.

A note on cost

All seven tools listed here are free or have generous free tiers sufficient for small to mid-size teams, which matters because sandboxing shouldn't be the thing a team skips due to budget pressure. The actual cost of setting this up is engineering time, a day or two for the minimal version described above, not licensing fees. Weighed against the cost of even one incident involving an assistant running something against the wrong environment, that setup time is one of the better returns available in this entire guardrails conversation.

Common mistakes teams make when setting this up

The most common mistake is seeding the sandbox from a production database dump instead of synthetic data, which defeats most of the point. A sandbox that contains real customer records is no longer a place where mistakes are free, it's just a second copy of the same risk you were trying to isolate. Faker or an equivalent synthetic data generator should be the default, and any exception, seeding from a scrubbed or anonymized production snapshot, should get the same scrutiny as granting access to production data anywhere else, because that's functionally what it is.

The second common mistake is treating the sandbox as a one-time setup rather than something that needs to stay current with schema changes. A scratch database seeded once at project start, and never refreshed, drifts from the real schema within a few months, and an assistant testing migrations against a stale sandbox will confidently generate changes that conflict with columns or constraints added since. Rebuilding the sandbox's schema, even if the synthetic data itself doesn't need refreshing as often, should be part of your regular migration or schema-change workflow, not a separate maintenance task that quietly falls off the list.

Measuring whether the sandbox is actually working

A sandbox that never catches anything might mean your assistant is unusually careful, or it might mean the sandbox isn't representative enough of real conditions to surface the failure modes that matter. Track how often a migration or change gets modified after sandbox testing but before it reaches a human reviewer, versus how often something slips through sandbox testing and gets caught in human review instead. If the second number is consistently higher than the first, the sandbox isn't earning its keep, and it's worth investing more in making it representative, more realistic data volume, more accurate constraint definitions, closer parity with the real database engine version, rather than treating it as a checkbox that's already been handled.

Putting it together

None of these tools individually solves the sandboxing problem. Together, Docker or Compose for isolation, Testcontainers or SQLite for a real database to test against, act for CI validation, and Faker for data that doesn't carry real customer risk, they cover most of what a small team needs before ever letting an assistant apply anything to a shared environment.

We cover the permission-tiering side of this problem, what an assistant should be allowed to execute once it has moved past the sandbox and into shared infrastructure, in 137Foundry's guide to AI coding assistant guardrails. The sandbox is tier two of that framework; these seven tools are what build it.

For teams setting this up from scratch, more on how we approach tooling and infrastructure decisions like this is at 137foundry.com.

What Actually Breaks When an AI Coding Assistant Has Too Much Shell Access

137Foundry — Fri, 10 Jul 2026 09:26:30 +0000

"Shell access" sounds like one permission, but in practice it's a bundle of very different capabilities that teams tend to grant all at once because splitting them feels like extra setup work. That bundling is where things go wrong, not because AI assistants are reckless, but because a single broad grant makes it hard to reason about what could actually happen when the assistant does something you didn't expect.

The bundle nobody unbundles

A typical "give the assistant shell access" setup lets it run tests, install dependencies, read files, write files, and execute arbitrary commands, all under the same permission. Each of those individually is low risk. Combined, they add up to something closer to "this process can do anything a developer with your credentials could do," which is a much bigger grant than most teams intend when they check the box.

# what "shell access" usually actually means in practice
npm test              # low risk
npm install <pkg>     # medium risk: supply chain exposure
rm -rf ./tmp           # fine, until the assistant's idea of "tmp" differs from yours
psql -f migration.sql  # high risk: no undo

The fix isn't revoking shell access entirely, it's granting the specific commands a workflow actually needs instead of a blanket shell.

Where the actual incidents come from

Most real incidents don't come from the assistant doing something it wasn't asked to do. They come from the assistant correctly executing a command whose scope was broader than the human intended. Someone asks it to "clean up temp files" and it interprets "temp" more broadly than expected. Someone asks it to "reset the local database" and the connection string in the environment pointed somewhere other than local, because an environment variable got left over from an earlier session.

# a pattern that prevents the second failure mode above
import os

def require_local_db():
    db_url = os.environ.get("DATABASE_URL", "")
    if "localhost" not in db_url and "127.0.0.1" not in db_url:
        raise RuntimeError(f"Refusing to run against non-local database: {db_url}")

A one-line guard like this, run before any destructive local command, catches an entire category of "wrong environment" incidents regardless of whether a human or an assistant issued the command.

Scoping shell access by task, not by tool

Rather than one broad shell grant, define specific allowed commands per task type. A test-running task gets npm test and nothing else. A dependency-update task gets npm install scoped to a lockfile diff a human reviews after. A migration-drafting task gets read access to the schema and write access to a migration file, but never execute access against anything but a disposable local database.

# example: scoping assistant permissions by task type
tasks:
  run_tests:
    allowed_commands: ["npm test", "npm run lint"]
  draft_migration:
    allowed_commands: ["npm run migrate:generate"]
    forbidden_commands: ["npm run migrate:apply"]
  update_deps:
    allowed_commands: ["npm install"]
    requires_human_review: true

This is more setup than granting one shell and calling it done, but it converts "trust the assistant to behave" into "the assistant physically cannot do the thing we don't want," which is a much sturdier guarantee. Sandboxing tools like Docker make this concrete: run the assistant's shell inside a container with no route to production credentials, and the scoping problem partly solves itself at the infrastructure layer instead of relying entirely on configuration discipline. Orchestration platforms like Kubernetes extend this further for teams running assistant sandboxes at scale, with network policies that can enforce the isolation boundary at the cluster level rather than trusting every container to be configured correctly by hand.

Why scoping by task beats scoping by tool

A common instinct is to scope permissions per tool, this assistant gets shell, this other integration gets API access, and call it done. That misses the actual risk surface, because the same tool used for two different tasks can have wildly different blast radius. A shell that can run npm test is fine for a hundred different task types. A shell that can also run npm run migrate:apply is not fine for almost any task type except the one specifically designed to apply reviewed migrations through a gated path. Scoping by tool treats these as the same grant. Scoping by task treats them correctly as different ones.

# a task-scoped config catches what a tool-scoped one misses
# BAD: one shell grant covers everything
assistant_permissions:
  shell: true   # now it can run npm test AND npm run migrate:apply

# BETTER: permissions attach to the task, not the tool
tasks:
  fix_failing_test:
    shell_commands: ["npm test", "npm run lint"]
  draft_schema_change:
    shell_commands: ["npm run migrate:generate"]
    forbidden: ["npm run migrate:apply", "npm run deploy"]

The credential leakage problem that's easy to miss

Even with commands properly scoped, a shell session often carries environment variables far broader than the task needs, database URLs for every environment, API keys for services the current task never touches, cloud credentials with permissions well beyond what a test run requires. An assistant with shell access to run tests, but whose environment happens to include a production database URL as an unused variable, has effectively been granted access to that database the moment any command in its session could reference that variable, intentionally or not.

# strip unrelated credentials before handing off a shell session
import os

ALLOWED_FOR_TEST_RUN = {"NODE_ENV", "CI", "DATABASE_URL_TEST"}

def scoped_env():
    return {k: v for k, v in os.environ.items() if k in ALLOWED_FOR_TEST_RUN}

This is a small amount of extra plumbing that closes off an entire class of "the assistant technically had access" incidents that never show up in an audit of granted permissions, because the permission was never explicitly granted, it was just sitting in an environment nobody scoped.

The migration case specifically

Database migrations are the sharpest example of why bundled shell access is dangerous, because they're one of the few operations in a typical stack that isn't cleanly reversible by reverting a commit. Reference material on the database side matters too. The PostgreSQL documentation covers exactly which DDL operations are transactional and reversible within a session, knowledge that should inform where you draw the disposable-versus-shared line for any given migration. We wrote a full breakdown of how to tier assistant permissions specifically around migrations, including a concrete pattern for keeping draft and apply as separate, separately-gated actions, in our guide to AI coding assistant guardrails for database migrations. The scoping principle there generalizes to shell access broadly: separate what the assistant can propose from what it can execute, and gate the second one on a human every time the action isn't trivially reversible.

A checklist worth running before granting shell access at all

Before wiring an AI coding assistant into any shell environment, walk through a short list: what specific commands does this task actually need, not what commands might be convenient to have available. Does the environment the shell runs in contain any credential broader than the current task requires. Is there a disposable version of whatever this command would touch, and if so, is the assistant actually pointed at that disposable version by default rather than by careful configuration someone has to remember every time. Is there a human or CI gate between "this command ran successfully" and "this change is live somewhere that matters."

None of these questions require sophisticated tooling to answer, they require someone to actually sit down and answer them before the first grant happens, rather than retroactively after an incident forces the question. Teams that do this upfront tend to end up with narrower, more specific shell configurations than teams that grant broadly and plan to tighten later, mostly because tightening later requires admitting the original grant was too broad, which is a harder conversation to have than getting the scope right from the start.

Why this is worth the setup time

Scoped shell access takes longer to configure than a single broad grant, there's no getting around that. But the alternative cost isn't zero, it's deferred and often invisible until the specific command that shouldn't have been possible actually runs. A well-scoped configuration, once built, is also reusable across projects and easier to reason about during a security review, since "here's exactly what this assistant can execute and why" is a much stronger answer than "it has shell access and we trust it," even when the assistant has, so far, behaved exactly as expected.

More on how we approach this kind of tooling setup at 137foundry.com.

How to Write a Data Export Script That Survives a Vendor Migration

137Foundry — Wed, 08 Jul 2026 09:17:46 +0000

A vendor's built-in export button is designed to satisfy a support ticket, not to power a migration. If you actually need your data portable, the safer move is owning a small export script yourself, one that runs on a schedule and gives you a known-good copy independent of whatever the vendor's UI decides to include.

The Requests library is the natural starting point for this in Python, since most vendor APIs are plain REST over HTTPS and Requests handles the pagination and retry patterns below with minimal boilerplate.

Step 1: Identify the real source of truth

Before writing anything, map which objects actually matter. Not every table in a vendor's schema is worth exporting, but skipping the wrong one means discovering a gap mid-migration. Prioritize anything with relational structure (deals linked to contacts, tickets linked to accounts) since that structure is what most built-in export buttons flatten away.

Step 2: Pull via the documented API, not the UI export

Most SaaS platforms expose a REST API with pagination and rate limits. Use that instead of the UI export feature, because API responses typically preserve IDs and relationships that a CSV export drops.

import requests
import time

def export_all(endpoint, headers, page_size=100):
    results = []
    cursor = None
    while True:
        params = {"limit": page_size}
        if cursor:
            params["cursor"] = cursor
        resp = requests.get(endpoint, headers=headers, params=params)
        resp.raise_for_status()
        data = resp.json()
        results.extend(data["items"])
        cursor = data.get("next_cursor")
        if not cursor:
            break
        time.sleep(0.2)  # respect rate limits
    return results

Step 3: Preserve relationships, not just fields

Store foreign keys alongside the records they reference, even if the new system will eventually need different IDs. You can remap IDs during a real migration. You can't reconstruct a relationship that was never captured in the first place. If you're storing exports as plain files rather than a database, a format like JSON Lines keeps each record independently parseable, which makes partial recovery much easier if a later export run gets interrupted partway through.

def export_with_relationships(deals, contacts):
    contact_lookup = {c["id"]: c for c in contacts}
    enriched = []
    for deal in deals:
        contact = contact_lookup.get(deal.get("contact_id"))
        enriched.append({
            **deal,
            "contact_email": contact["email"] if contact else None,
            "contact_name": contact["name"] if contact else None,
        })
    return enriched

This kind of denormalization at export time costs almost nothing computationally and saves enormous effort later, since you won't need the original vendor's system alive just to resolve a foreign key.

Step 4: Run it on a schedule, before you need it

Set this script up as a weekly or monthly job, storing exports somewhere durable like object storage. The value isn't the export itself, it's having a recent, known-good copy on the day you actually decide to leave, instead of racing a support ticket during a contract deadline.

A simple cron entry or scheduled cloud function is enough for most teams. The frequency matters less than the consistency. A monthly export you can trust beats a "we'll figure it out when we need it" plan every time, because the day you actually need it is rarely a day when the vendor relationship is going well.

Step 5: Validate the export by reconstructing a record

Periodically pick one real record and try to fully reconstruct it from your export alone. If you can't, your export is missing something, and it's much cheaper to find that gap during a routine check than during an actual migration crunch.

Automate this validation where you can. A script that spot-checks a handful of exported records against the live system's API, comparing field-by-field, will catch schema drift long before a human reviewer would notice the export quietly stopped capturing a field the vendor added last quarter.

Step 6: Keep the schema documented alongside the export itself

The export script's own output format needs documentation too, or you're just relocating the same problem one layer down. A README next to the export bucket describing what each field means, which fields are foreign keys, and which fields changed meaning over time, turns a pile of JSON files into something an unfamiliar engineer could actually use during a real migration six months from now, possibly without you in the room.

Step 7: Handle pagination and rate limits like they're permanent, not incidental

It's tempting to write a quick script that assumes the API will always return everything in a single page, especially when your dataset is small today. Don't. Datasets grow, and a script that silently truncates results once you cross a pagination threshold is worse than no script at all, because it creates false confidence in an export that's actually incomplete.

def export_all_paginated(base_url, headers, page_size=100, max_retries=3):
    all_items = []
    cursor = None
    while True:
        for attempt in range(max_retries):
            try:
                resp = requests.get(
                    base_url,
                    headers=headers,
                    params={"limit": page_size, "cursor": cursor} if cursor else {"limit": page_size},
                    timeout=30,
                )
                resp.raise_for_status()
                break
            except requests.RequestException:
                if attempt == max_retries - 1:
                    raise
                time.sleep(2 ** attempt)
        data = resp.json()
        all_items.extend(data["items"])
        cursor = data.get("next_cursor")
        if not cursor:
            return all_items

Building in retries and exponential backoff from the start means the script keeps working reliably as your data volume grows, rather than needing a rewrite the first time it hits a rate limit mid-export.

Step 8: Store more history than you think you need

Keep several months of prior exports, not just the latest one. If you discover a data quality issue, whether it's a vendor bug or something in your own script, having historical exports lets you pinpoint when the problem started rather than only knowing it exists in the current snapshot. Object storage is cheap enough that this is rarely a meaningful cost tradeoff against the diagnostic value it provides. A standard object store like Amazon S3 or an equivalent from another cloud provider is more than sufficient for this, and versioned buckets give you the historical retention almost for free.

Photo by Umberto on Unsplash

Why this is worth doing even if you never migrate

Even teams with no near-term plans to switch vendors get real value from this pipeline. It becomes a natural audit trail, a disaster-recovery hedge if the vendor ever has a serious outage or data loss incident, and a source of truth you can query directly for analysis without hitting API rate limits against the live system. The migration-readiness benefit is real, but it's often the smallest of the three reasons teams end up glad they built this.

Step 9: Monitor the export job itself, not just the vendor

A scheduled export script that silently fails is arguably worse than no script at all, since it creates false confidence that a safety net exists when it actually stopped working months ago. Wire basic monitoring into the job: an alert if it fails to run, and a sanity check comparing the new export's record count against the previous run's count. A sudden, unexplained drop in record count is often the first sign something changed on the vendor's side, whether that's an API contract change, a permissions issue, or a genuine data loss event worth investigating immediately.

Step 10: Treat schema changes on the vendor's side as expected, not exceptional

Vendors change their APIs. Fields get renamed, deprecated, or restructured, usually with advance notice buried in a changelog nobody on your team reads regularly. Build your export script defensively, so a missing field logs a warning rather than crashing the entire job, and review those warnings periodically rather than only discovering a schema change when someone actually needs the missing data during a real migration.

Putting it together

None of these ten steps require exotic tooling or a dedicated team. A single well-monitored script, running on a schedule, storing structured exports with preserved relationships in durable storage, gets most teams most of the portability benefit that otherwise depends entirely on trusting a vendor's own export feature to be complete. The upfront investment is measured in hours, not weeks, and it pays for itself the first time a migration decision needs to happen faster than a vendor's support queue can accommodate.

This kind of proactive export discipline is exactly the countermeasure to the lock-in risk we break down in our framework for evaluating vendor lock-in before you sign a SaaS contract. Data portability is one of four dimensions worth scoring, and owning your own export pipeline is the practical way to keep that score in your favor regardless of what the vendor's contract says.

If you're building this kind of pipeline and want a second set of eyes on the schema design, that's the kind of data integration work we do often.

How to Audit Your Own Integration Surface Before It Becomes a Lock-In Problem

137Foundry — Wed, 08 Jul 2026 09:17:44 +0000

Vendor lock-in doesn't usually happen because of one big decision. It accumulates from dozens of small ones: a webhook here, a Zapier automation there, a script someone wrote during an on-call incident that quietly became load-bearing. None of those individual choices look risky at the time. The aggregate is what hurts.

Photo by Dimitri Karastelev on Unsplash

Why nobody notices until it's expensive

Each integration is built by whoever needed it, when they needed it, with no visibility into what else already depends on the same vendor. Six months later, that vendor is a single point of failure for five different workflows and nobody who built any single one of them realizes the total blast radius.

The fix isn't stopping people from building integrations. It's making the total integration surface visible, on a cadence, so the dependency count is a known number instead of a surprise.

This is the same failure mode that shows up in technical debt generally: no single decision looks unreasonable in isolation, but the aggregate becomes a genuine liability nobody explicitly chose. The difference with vendor integrations is that the debt isn't just harder-to-maintain code, it's negotiating leverage you've quietly handed to a vendor whose renewal pricing gets more aggressive every year you can't easily leave.

Why this matters more for smaller teams, not less

There's an intuition that lock-in is mostly an enterprise problem, since large companies have more systems and more integration points. In practice, smaller teams are often more exposed, because they lack a platform or infrastructure team whose job is specifically to track this kind of dependency. A twenty-person startup can accumulate the same integration sprawl as a five-hundred-person company, just with fewer people around who'd notice it happening.

A simple audit you can run this week

Start with the vendor's own developer console if it has one. Most platforms expose a list of active webhooks, API keys, and connected apps under account or security settings. That list is your starting inventory.

Cross-reference it against your automation platform (Zapier, Make, n8n, or an internal script repo). For every integration you find, capture three things: what triggers it, what it does, and who would notice if it silently broke tomorrow. That third question is the one teams skip, and it's the one that matters most when you're planning a migration.

# a minimal starting point for tracking integration dependencies
integrations = [
    {"vendor": "acme_crm", "trigger": "deal.won", "action": "sync_to_billing", "owner": "finance"},
    {"vendor": "acme_crm", "trigger": "contact.updated", "action": "update_mailing_list", "owner": "marketing"},
]

def blast_radius(vendor_name):
    return [i for i in integrations if i["vendor"] == vendor_name]

Even a spreadsheet version of this is enough to start. The goal is just converting invisible dependency count into a visible one.

Turning the audit into an ongoing habit

Run this audit before adopting any new tool that another system might eventually integrate with, and again roughly every quarter for vendors you already depend on heavily. Integration count only grows over time; catching it early is cheaper than discovering it during a forced migration.

Assign an actual owner to this recurring audit, not just a shared responsibility that quietly belongs to nobody. Quarterly reviews that everyone agrees are important but nobody specifically owns tend to slip the first time a sprint gets busy, and the integration count that would've taken fifteen minutes to review in March becomes a much bigger surprise by September.

A more detailed inventory template

Once you've got a first pass done, formalize it slightly. For each integration, capture the vendor, the trigger event, the downstream action, the business owner, and a rough severity if it silently broke: cosmetic, annoying, or genuinely business-critical. That severity column is what turns a flat list into something you can actually prioritize during a migration planning conversation.

integration_inventory = [
    {
        "vendor": "acme_crm",
        "trigger": "deal.won",
        "action": "sync_to_billing",
        "owner": "finance",
        "severity": "critical",
    },
    {
        "vendor": "acme_crm",
        "trigger": "contact.updated",
        "action": "update_mailing_list",
        "owner": "marketing",
        "severity": "annoying",
    },
]

def critical_dependencies(vendor_name):
    return [
        i for i in integration_inventory
        if i["vendor"] == vendor_name and i["severity"] == "critical"
    ]

A vendor with two critical dependencies and six cosmetic ones is a very different migration project than a vendor with eight critical dependencies, even if the raw integration count looks similar on paper.

What to do once you know the number

Finding out you have eighteen integrations against one vendor isn't itself a crisis. It's information. Some of those integrations might be trivial to replace, others might genuinely justify staying with the vendor even at a worse price, because the switching cost outweighs the savings. The point of the audit isn't to force a migration, it's to make sure that decision gets made deliberately, with real numbers, instead of by default because nobody had visibility into the total picture.

Where this becomes genuinely actionable is at renewal time. A vendor that knows you've quietly become dependent on eighteen integrations negotiates very differently than one who knows you've tracked that number and have a documented plan for the ones that matter least. Visibility into your own integration surface is, in a very literal sense, negotiating leverage.

We cover the broader risk framework, including data portability and contract exit terms alongside integration depth, in our guide to evaluating vendor lock-in risk before committing to a platform. Integration surface is one of four dimensions worth scoring before you sign, not after.

For teams standardizing how services talk to each other, the OpenAPI Initiative is worth a look, since documented, standard API contracts make future integration audits considerably less painful than reverse-engineering an undocumented one. The Zapier platform is also a reasonable place to centralize visibility if your team's automations are scattered across several no-code tools already, since it at least puts everything in one dashboard rather than five. More on how we approach this kind of work at 137foundry.com.

A pattern worth watching for specifically

One integration type deserves extra scrutiny during the audit: anything where a vendor's data flows into a system of record, meaning a place where the data becomes the canonical version rather than just a cached copy. A read-only dashboard pulling from a vendor's API is low risk, since you can rebuild the dashboard without touching the underlying data. A workflow where a vendor's webhook writes directly into your billing system or your customer database is a different category of risk entirely, because untangling that dependency means carefully verifying data integrity on both sides during the transition, not just rewiring a data source.

Flag these system-of-record integrations distinctly in your audit. They're usually a small fraction of the total integration count but represent a disproportionate share of the real migration risk, and they're exactly the ones worth prioritizing if you ever do decide a vendor relationship needs to end.

Making the audit sustainable long-term

The biggest risk to this whole practice isn't that teams disagree it's valuable, it's that the audit quietly stops happening after the first one or two rounds, once the initial enthusiasm fades and other priorities crowd it out. Tying it to an existing recurring ritual, a quarterly planning session or an existing infrastructure review, tends to work better than treating it as a standalone calendar reminder that's easy to snooze indefinitely. The goal is a habit that survives turnover on the team, not a one-time project that only exists because one person happened to care about it this quarter.

Why Faceted Navigation Is the Silent Killer of Crawl Budget on Ecommerce Sites

137Foundry — Tue, 07 Jul 2026 09:34:54 +0000

Faceted navigation is a genuinely good user experience feature. Letting shoppers filter by size, color, brand, and price at the same time is table stakes for any catalog with more than a few dozen products. It's also one of the most reliable ways to accidentally generate tens of thousands of crawlable URLs from a single category page.

The math nobody does until it's a problem

A category page with five facets, each offering four to six options, doesn't produce five to thirty variations. It produces every mathematically possible combination of selected filters, which climbs into the thousands fast. Multiply that across every category on the site, and a catalog with a few hundred real products can generate URLs numbering in the hundreds of thousands.

Each of those is a request Googlebot might make. Crawl budget, the finite number of requests a search engine is willing to spend on your domain per day, isn't infinite, and every request spent on ?color=red&size=large&brand=x is a request not spent discovering or refreshing content that actually matters.

Why this is worse for ecommerce specifically

Product availability and pricing change constantly. If crawl budget is being consumed by filter permutations, Googlebot revisits your actual product pages less often, which means stale pricing and out-of-stock listings can linger in search results longer than they should. That's a direct hit to conversion rate from organic traffic, not just an abstract ranking concern.

The architecture decision hiding inside this problem

The real question isn't "how do we stop this," it's "which filter combinations are worth their own indexable page." A single-facet filter on a popular category, ?color=red on a shirts page, might represent real search demand worth capturing with its own optimized template. A four-facet combination almost certainly doesn't.

This means the fix isn't purely technical. It requires someone to look at actual search query data and decide, facet by facet, what earns an indexable page versus what gets canonicalized away or blocked outright. Google's own documentation on faceted navigation best practices covers the mechanics of the canonical and parameter handling options in detail, but the demand analysis is a judgment call specific to your catalog.

A pragmatic default for teams without time for the full analysis

If you don't have the bandwidth to run a full search-demand study per facet, a reasonable default is: index single-facet category and subcategory pages if they get meaningful search volume, canonicalize everything else back to the base category page, and block pure tracking or session parameters at the robots.txt level. It's not perfect, but it stops the bleeding while a more deliberate strategy gets built.

Why fixing it later costs more than fixing it now

Parameter sprawl compounds. Every month a faceted navigation system runs without canonical tags or indexing rules, more of those combinations get discovered and indexed by search engines, and more of them accumulate whatever thin backlink or internal link signal happens to reach them by accident. Cleaning up a six-month-old mess is a bounded project. Cleaning up a four-year-old one on a catalog that's grown ten times larger in that time is a much bigger undertaking, both because there's more to classify and because some of those stray indexed pages will have picked up real search traffic that needs to be redirected carefully rather than just blocked.

This is one of the clearer cases in technical SEO where the cost curve is genuinely exponential rather than linear. Catching it in the first few months after a faceted navigation feature ships is dramatically cheaper than catching it after the catalog has doubled twice.

Signals that your faceted navigation specifically is the culprit

Not every duplicate content problem traces back to facets, so it's worth confirming before assuming that's the fix needed. Check whether the base paths showing the most parameter variance in your crawl or log data correspond to your filterable listing pages specifically, versus other sources like session tracking or CMS-generated print views. If the worst offenders are /category/* or /products/* style routes with multiple query parameters stacked together, faceted navigation is very likely the primary driver, and the fix belongs at the filter component level rather than scattered across individual page templates.

The engineering side people underestimate

It's tempting to treat this as purely an SEO team's problem to flag and a frontend team's problem to fix in isolation, but the cleanest implementations come from the two working from the same source of truth about which facet combinations are canonical, indexable, or blocked. If SEO hands over a spreadsheet of rules and engineering implements them once without a shared mechanism for keeping them in sync, the rules drift the next time a facet gets added or renamed. A small config file or database table mapping facet combinations to their intended indexing treatment, referenced by both the sitemap generator and the canonical tag logic, tends to hold up much better over time than rules baked separately into multiple places.

Framework and platform defaults are not a substitute for checking

It's worth explicitly checking rather than assuming, even on platforms with a reputation for handling this well out of the box. A platform might correctly canonicalize single-facet filters by default while leaving multi-facet combinations entirely unhandled, which is exactly the gap that causes the most damage since multi-facet combinations are also the ones generating the largest URL counts. Read your specific platform or framework's documentation on faceted navigation and canonical handling directly rather than assuming "it's a mature platform, it must handle this," since defaults vary significantly and change between versions.

Measuring whether the fix actually helped

After deploying canonical tags and indexing rules across a faceted navigation system, the metric to watch isn't just the Search Console duplicate content count, though that should trend down. Crawl stats for your actual product and category pages, the ones that matter for conversions, should show increased crawl frequency over the following month as budget that was previously spent on facet combinations gets redirected toward pages Google now understands are worth revisiting more often. If that shift doesn't happen, the fix likely didn't cover enough of the parameter surface area, and it's worth re-running the log analysis to check for combinations that were missed.

What this looks like in practice

We walked through the full sequence, auditing scope with Search Console and log data, classifying parameters into content-changing versus non-content buckets, and applying canonical tags correctly, over at 137foundry.com. The ecommerce case is really just that same process applied at a larger scale, since faceted navigation is the single biggest parameter generator most catalog sites have.

If you're evaluating platforms, Shopify's own developer documentation and most major ecommerce platforms document how their filter systems handle canonical tags by default, and it's worth checking before assuming a fix is needed. Some platforms already handle single-facet canonicalization correctly out of the box and only fall short on multi-facet combinations. For a platform-agnostic reference on how crawlers evaluate faceted URLs generally, Google's Search Central crawling and indexing documentation is the most current source, since platform vendor docs sometimes lag behind changes in how search engines actually treat these patterns.

Getting the crawl budget spent in the right place is one of those problems that compounds quietly. A catalog that fixes this early scales cleanly. One that doesn't ends up doing a much larger cleanup years later once the parameter sprawl has had time to multiply.

How to Write a Script That Finds Every Parameterized URL Variant on Your Site

137Foundry — Tue, 07 Jul 2026 09:33:40 +0000

Before you can fix duplicate content caused by URL parameters, you need to know the actual scope. Guessing from a few examples in Search Console misses most of the problem. This walks through a small Python script that parses server logs and groups requests by their base path, so you can see exactly how many parameter variants each route is generating.

Why log parsing beats Search Console alone

Search Console tells you which duplicate issues Google noticed, but it samples and aggregates, so smaller-volume parameter combinations don't always surface individually. Your own server logs (or a CDN's access logs) contain every request that actually happened, which gives you the real count.

Setting up the parser

This assumes a standard combined log format, common to Apache and nginx by default:

import re
from collections import defaultdict
from urllib.parse import urlparse, parse_qs

LOG_PATTERN = re.compile(
    r'"(?:GET|POST) (?P<path>[^\s]+) HTTP'
)

def parse_log_line(line):
    match = LOG_PATTERN.search(line)
    if not match:
        return None
    return match.group("path")

def group_by_base_path(log_file):
    groups = defaultdict(set)
    with open(log_file, "r", encoding="utf-8", errors="ignore") as f:
        for line in f:
            path = parse_log_line(line)
            if not path:
                continue
            parsed = urlparse(path)
            base = parsed.path
            query = parse_qs(parsed.query)
            if query:
                param_signature = tuple(sorted(query.keys()))
                groups[base].add(param_signature)
    return groups

Reading the results

Once you've grouped requests, sort by the number of distinct parameter signatures per base path to find your worst offenders:

groups = group_by_base_path("access.log")
ranked = sorted(groups.items(), key=lambda kv: len(kv[1]), reverse=True)

for base_path, signatures in ranked[:20]:
    print(f"{base_path}: {len(signatures)} distinct parameter combinations")

A route showing forty or fifty distinct parameter combinations against a handful of actual product variations is a clear candidate for canonical tag cleanup, and probably a robots.txt rule for anything that's pure tracking noise.

Extending it to check canonical tags directly

Once you have a list of the worst offending base paths, cross-reference a sample of each parameter variant's actual canonical tag using a lightweight request:

import requests
from bs4 import BeautifulSoup

def get_canonical(url):
    resp = requests.get(url, timeout=10)
    soup = BeautifulSoup(resp.text, "html.parser")
    tag = soup.find("link", rel="canonical")
    return tag["href"] if tag else None

Run this against ten or so parameter variants per base path and confirm they all resolve to the same canonical URL. Inconsistent results here point to a templating bug rather than a missing rule, which is a different fix.

Filtering out bot traffic that isn't Googlebot

Not every request in your logs represents crawl budget you care about optimizing. Scrapers, monitoring services, and other bots hit the same URLs, and including them in your analysis inflates the apparent problem or, worse, hides the real Googlebot pattern underneath noise from unrelated traffic. Filter by user agent before counting:

def is_googlebot(line):
    return "Googlebot" in line

def group_googlebot_only(log_file):
    groups = defaultdict(set)
    with open_log(log_file) as f:
        for line in f:
            if not is_googlebot(line):
                continue
            path = parse_log_line(line)
            if not path:
                continue
            parsed = urlparse(path)
            query = parse_qs(parsed.query)
            if query:
                signature = tuple(sorted(query.keys()))
                groups[parsed.path].add(signature)
    return groups

A simple user agent string match is a reasonable starting filter, though be aware that user agent strings can be spoofed. If you need to confirm requests are genuinely from Google's crawlers rather than something pretending to be, a reverse DNS lookup against the requesting IP, checked against Google's published IP ranges, is the more rigorous approach, but for a first-pass audit the string match is usually good enough to get directionally accurate numbers.

Handling gzipped and rotated logs

Most production setups rotate and compress logs after a day or two, so a real version of this script needs to handle .gz files transparently:

import gzip

def open_log(path):
    if path.endswith(".gz"):
        return gzip.open(path, "rt", encoding="utf-8", errors="ignore")
    return open(path, "r", encoding="utf-8", errors="ignore")

Swap this into group_by_base_path in place of the plain open() call, and loop over every rotated file in your log directory rather than a single file, so you get a representative sample across at least a week rather than a few hours that might not reflect a weekly cron job or a scheduled crawl pattern.

Adding a simple CLI wrapper for repeated runs

If you expect to run this audit more than once, wrapping it as a small command-line tool with argparse saves time on every subsequent run:

import argparse

def main():
    parser = argparse.ArgumentParser(description="Find parameterized URL variants from access logs")
    parser.add_argument("log_dir", help="Directory containing log files (supports .gz)")
    parser.add_argument("--top", type=int, default=20, help="Number of worst offenders to show")
    parser.add_argument("--googlebot-only", action="store_true", help="Filter to Googlebot requests only")
    args = parser.parse_args()
    # wire up group_by_base_path / group_googlebot_only based on args here
    print(f"Analyzing logs in {args.log_dir}, showing top {args.top} offenders")

if __name__ == "__main__":
    main()

This turns a one-off analysis script into something a whole team can rerun monthly as part of a standing technical SEO check, rather than something only one person remembers how to invoke correctly.

Exporting results for non-technical teammates

The raw script output is fine for an engineer, but an SEO specialist or product manager on the same team will get more value from a CSV they can sort and filter themselves. Adding an export step takes a few lines:

import csv

def export_to_csv(ranked_results, output_path="parameter_audit.csv"):
    with open(output_path, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["base_path", "distinct_combinations", "total_requests"])
        for base_path, data in ranked_results:
            writer.writerow([base_path, len(data), sum(data.values()) if isinstance(data, dict) else ""])

Sharing this file alongside the raw numbers turns a script only one engineer can run into a shared artifact the whole team can reference when deciding which templates to prioritize fixing first.

Turning the output into a prioritized list

Raw counts of parameter combinations are useful, but pairing them with actual crawl frequency data makes the prioritization sharper. Extend the script to also track how many total requests hit each base path, not just how many distinct combinations exist:

def group_with_counts(log_file):
    groups = defaultdict(lambda: defaultdict(int))
    with open_log(log_file) as f:
        for line in f:
            path = parse_log_line(line)
            if not path:
                continue
            parsed = urlparse(path)
            base = parsed.path
            query = parse_qs(parsed.query)
            signature = tuple(sorted(query.keys())) if query else ()
            groups[base][signature] += 1
    return groups

Now you can rank base paths not just by how many distinct combinations exist, but by total wasted request volume, which is the number that actually matters for crawl budget. A path with fifteen combinations getting hit constantly is a bigger problem than one with sixty combinations that barely get crawled at all.

Where this fits in the bigger fix

This script gets you the diagnostic data. The actual remediation, classifying parameters and applying canonical tags correctly, is covered step by step in 137foundry.com's guide to stopping URL parameters from creating duplicate content. Running this kind of audit before touching any template code saves you from fixing the wrong ten pages while the real offenders keep piling up, since intuition about which routes are the worst offenders is frequently wrong until you actually measure it.

The Python urllib.parse documentation covers the parsing utilities used here in more depth if you need to handle edge cases like array-style parameters or encoded query strings. If you're working with a larger log volume than fits comfortably in memory, the pandas documentation covers chunked reading patterns that scale better than the dictionary-based approach shown here once you're past a few million log lines. For confirming which IP ranges genuinely belong to Google's crawlers before trusting a user-agent-based filter for anything beyond a rough first pass, Google's published crawler IP list documents the verification process directly.

DEV Community: 137Foundry

How to Run Old and New Systems in Parallel Without Doubling Your Workload

Step 1: Feed both systems the same inputs automatically

Step 2: Automate the comparison, don't do it by eye

Step 3: Triage discrepancies by pattern, not by individual occurrence

Step 4: Set an explicit exit criteria before you start

Step 5: Cover a full business cycle if the system has one

Handling discrepancies that turn out to be the legacy system's bug

What this catches that a code review can't

References

Deciding when a parallel run isn't worth building

A note on team morale during the validation period

7 Free Tools for Auditing an Old Codebase Before You Migrate It

1. Your version control's blame and log commands

2. A static analysis tool for dead code detection

3. A dependency graph visualizer

4. A test coverage report, even a rough one

5. A database schema visualizer

6. A log aggregator, even a temporary one

7. A documentation generator for whatever comments do exist

Putting these together with the human side of the audit

A note on scope

Combining tool output with human context

Keeping the audit from becoming its own multi-month project

How to Write a CLAUDE.md or .cursorrules File That Actually Gets Followed

Step 1: Audit What You're Actually Repeating in Chat

Step 2: Separate Durable Rules From Temporary Ones

Step 3: Point at Examples Instead of Describing Patterns Abstractly

Step 4: Add an Explicit "Ask First" List

Step 4b: Test the Rules Against a Real Task

Step 5: Keep It Under a Page and Revisit It in Pull Requests

Step 6: Treat It Like Any Other Documentation Debt

Where This Fits With the Rest of Your Tooling

A Realistic Timeline

When to Consider It Done

7 Free Tools for Auditing What Your AI Coding Assistant Actually Changed

1. Git Itself

2. GitHub Pull Request Review Tools

3. ESLint

4. Prettier

5. Static Analysis via SonarQube or Similar

6. Vitest or Your Existing Test Runner, Run on Every AI-Assisted Commit

7. A Simple Changelog Convention

Putting These Together Instead of Picking One

How Often Teams Skip This and Regret It

Making the Habit Stick Past the First Month

What This Looks Like Applied Consistently

What Actually Happens When You Paginate a Live, Growing Table

The mental model that causes the bug

A concrete walkthrough

Why this is worse than it sounds

The fix: stop asking "skip N," start asking "after this point"

The tiebreaker is not optional

Testing for this specific failure mode

A variant of the same bug: deletions instead of insertions

The broader lesson

A checklist before you ship any paginated endpoint

How to Build API Pagination That Survives Rate Limits and Client Retries

Step 1: Make the cursor idempotent, not session-based

Step 2: Return rate limit headers on every paginated response

Step 3: Make retries safe by design, not by convention

Step 4: Implement exponential backoff on the client side

Step 5: Set a reasonable page size ceiling to reduce total request volume

Step 6: Log cursor state on failure for debugging and safe resumption

Step 7: Version the cursor format so future changes don't break old clients

Step 8: Handle partial failures in bulk consumers gracefully

Putting it together

Further reading

Seven Free Tools for Sandboxing What an AI Coding Assistant Can Execute

1. Docker

2. Docker Compose

3. Testcontainers

4. SQLite

5. act

6. Firecracker

7. Faker libraries (Faker.js / Python Faker)

Choosing between these based on team size

What a minimal setup actually looks like end to end

Extending the setup as the project grows

A note on cost