<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Prajakta Shende</title>
    <description>The latest articles on DEV Community by Prajakta Shende (@prajaktashende).</description>
    <link>https://dev.to/prajaktashende</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4199710%2Fbcf37357-f755-4f81-9f11-20e4cf61cf5b.png</url>
      <title>DEV Community: Prajakta Shende</title>
      <link>https://dev.to/prajaktashende</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/prajaktashende"/>
    <language>en</language>
    <item>
      <title>Pressure-Test Your Distributed Database Before Production Does It for You</title>
      <dc:creator>Prajakta Shende</dc:creator>
      <pubDate>Wed, 04 Nov 2026 23:10:14 +0000</pubDate>
      <link>https://dev.to/prajaktashende/pressure-test-your-distributed-database-before-production-does-it-for-you-2019</link>
      <guid>https://dev.to/prajaktashende/pressure-test-your-distributed-database-before-production-does-it-for-you-2019</guid>
      <description>&lt;p&gt;&lt;em&gt;By Prajakta Shende — Database Customer Engineer, Google Cloud, with close to a decade at AWS in the data space before that.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Nobody's database fails during the demo. They fail on Black Friday, on quarter-end close, or at 2 a.m. when a node loses its disk and the cluster has to decide — in seconds, with real customer traffic — whether it still has a quorum. If you haven't rehearsed that moment on purpose, the first rehearsal will be a production incident.&lt;/p&gt;

&lt;p&gt;Pressure testing isn't a benchmark. Benchmarks are marketing. Pressure testing is a dress rehearsal for the worst day of the year: it tells you where the cliff is, how the system behaves at the edge, and what recovery actually looks like — before your customers find out for you. Here's the checklist I wish someone had handed me a decade ago.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Test the workload you actually have, not the one in the benchmark
&lt;/h2&gt;

&lt;p&gt;Every database vendor ships numbers run on pristine schemas and uniform key distributions. Your production workload has neither. The two things that make or break a pressure test are the read/write ratio and the key skew — and both have to come from your real traffic.&lt;/p&gt;

&lt;p&gt;A 95/5 read-heavy workload behaves nothing like a 50/50 one. Skewed access (a handful of hot keys taking most of the traffic) stresses very different machinery than uniform access. Capture a representative slice of production queries — anonymized — and replay that. If you can't replay production traffic, build a synthetic mix with the same ratios and skew, and document your assumptions so the next person knows what's a guess. The benchmark number you quote in a design doc should be &lt;em&gt;your&lt;/em&gt; workload's number, not the vendor's.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Warm it up like production
&lt;/h2&gt;

&lt;p&gt;Cold caches lie. Every test that starts against an empty buffer pool, an unwarmed OS page cache, or a freshly restarted node will report performance that no production system will ever see again. Run your workload until the metrics stabilize — the same p99 you measured at minute 5 should still be there at minute 50. If it isn't, you're measuring warmup, not capacity.&lt;/p&gt;

&lt;p&gt;This sounds obvious. Nearly every "surprising" post-launch regression I've investigated skipped it.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Measure the tail, not the average
&lt;/h2&gt;

&lt;p&gt;Averages are how databases pass tests and fail customers. At scale, what your users feel is the tail: p99, p999, and the &lt;em&gt;shape&lt;/em&gt; of the tail under stress. Two systems with identical average latency can have wildly different p99 — and the one with the fatter tail is the one that pages you at 2 a.m.&lt;/p&gt;

&lt;p&gt;Log full latency histograms, not summaries. And measure the tail &lt;em&gt;during&lt;/em&gt; the ugly parts of the test: during a node failure, during a rebalance, during a rolling restart. The steady-state p99 is a vanity metric; the failure-mode p99 is the one your incident reviews will be about.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Kill things on purpose
&lt;/h2&gt;

&lt;p&gt;This is the part most teams postpone until production does it for them. A pressure test that never loses a node is a benchmark with extra steps. Deliberately kill processes, kill whole nodes, partition the network, stall a disk — and measure two things each time:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Detection time:&lt;/strong&gt; how long before the cluster notices and reacts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recovery behavior:&lt;/strong&gt; what happens to write availability, read latency, and consistency &lt;em&gt;during&lt;/em&gt; the window.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Write down what you expected before you run it. The gap between the expected recovery and the actual one is the most valuable output of the entire exercise — that gap is your incident playbook's first draft. I've seen clusters that were theoretically highly available take minutes to elect a new leader while the application timed out everything in flight. Theory doesn't pay the pager duty bill; measured recovery time does.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Find the cliff, don't just admire the slope
&lt;/h2&gt;

&lt;p&gt;Load testing usually stops at "it handled our target." That's half the job. Keep going past the target until the system degrades — and watch &lt;em&gt;how&lt;/em&gt; it degrades. Does latency rise linearly (you can plan around that), or does it fall off a cliff at some threshold (you cannot)?&lt;/p&gt;

&lt;p&gt;The cliff's location is the number your capacity planning actually needs. Run the test at 50%, 80%, 100%, 120% of expected peak and map the curve. The difference between "graceful degradation" and "catastrophic collapse" decides whether your on-call gets a warning or a wake-up call.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Soak it
&lt;/h2&gt;

&lt;p&gt;Some failures only appear with time: memory leaks, compaction debt, disk fragmentation, connection churn. A four-hour test won't find them. A 48-hour soak at realistic load will. It's boring, it's expensive to run, and it's the test that catches the bugs that show up in week three of production.&lt;/p&gt;

&lt;p&gt;Schedule it at least once before every major launch. If you can't afford the 48 hours, you definitely can't afford the week-three incident.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Validate consistency while things break, not just after
&lt;/h2&gt;

&lt;p&gt;It's not enough that the database comes back up. Verify that the &lt;em&gt;data&lt;/em&gt; is right: run a consistency checker against a known-good baseline while failures are being injected, not just once everything is calm. The scariest bugs I've seen weren't outages — they were systems that stayed up and served slightly wrong answers. Checksums, row counts, and aggregate comparisons during chaos testing catch the corruption that uptime metrics hide.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pre-production checklist
&lt;/h2&gt;

&lt;p&gt;Before any distributed database takes production traffic, I want written answers to five questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;What is our workload's real read/write ratio and key skew, and what did we replay?&lt;/li&gt;
&lt;li&gt;What is the measured p99 under failure (node loss, partition), not just steady state?&lt;/li&gt;
&lt;li&gt;What is the detection time and recovery time for a lost node, in numbers?&lt;/li&gt;
&lt;li&gt;Where is the cliff — at what multiple of expected peak does the system stop degrading gracefully?&lt;/li&gt;
&lt;li&gt;Did we run a 48-hour soak, and what did the tail look like at hour 48 vs. hour 4?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Pressure testing doesn't prevent incidents. It converts surprise incidents into rehearsed ones — and rehearsed incidents are the kind you survive without a postmortem titled "we didn't know."&lt;/p&gt;

</description>
      <category>cloud</category>
      <category>distributedsystems</category>
      <category>testing</category>
      <category>database</category>
    </item>
    <item>
      <title>What Breaks When You Migrate Petabytes: A Field Guide</title>
      <dc:creator>Prajakta Shende</dc:creator>
      <pubDate>Thu, 22 Oct 2026 07:42:32 +0000</pubDate>
      <link>https://dev.to/prajaktashende/what-breaks-when-you-migrate-petabytes-a-field-guide-4g4n</link>
      <guid>https://dev.to/prajaktashende/what-breaks-when-you-migrate-petabytes-a-field-guide-4g4n</guid>
      <description>&lt;p&gt;&lt;em&gt;By Prajakta Shende — Database Customer Engineer at Google Cloud, with close to a decade at AWS in the data space before that.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Every migration plan works on the whiteboard. I've spent a decade moving production databases — petabytes of them, across engines, clouds, and more cutover nights than I care to count — and I can tell you exactly where the plan meets reality. Petabytes don't break migrations in random places. They break them in five predictable ones, in roughly this order.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. The throughput math nobody does
&lt;/h2&gt;

&lt;p&gt;Before any dates get committed, three numbers decide your migration: how fast the source can be read, how fast the network can move data, and how fast the target can write. Your actual migration speed is the &lt;em&gt;slowest&lt;/em&gt; of the three — and it's almost never the one the plan assumed.&lt;/p&gt;

&lt;p&gt;The second mistake is treating the bulk copy as the schedule. It isn't. The cutover window is set by change-data-capture lag: how long it takes the target to replay every change that landed during the copy. On a quiet system that's minutes. On a system writing thousands of transactions a second, an untested CDC pipeline can fall hours behind — and you find out during the cutover window you promised the business.&lt;/p&gt;

&lt;p&gt;Do the math before you promise the date. Model the backfill, measure the change rate at peak (not average), and rehearse the catch-up. If you can't state your expected CDC lag at cutover within an order of magnitude, you don't have a plan yet. You have a hope.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Schema drift and the long tail of types
&lt;/h2&gt;

&lt;p&gt;Automated schema converters are wonderful liars. They'll convert 95% of your tables cleanly and report success — and the remaining 5% will eat your entire schedule.&lt;/p&gt;

&lt;p&gt;The failures are rarely dramatic. They're semantic: a timestamp type that silently drops timezone behavior, a collation difference that changes sort order and breaks a dedup query, numeric precision that rounds differently in the new engine. Nothing crashes. The data is just &lt;em&gt;wrong&lt;/em&gt;, quietly, in ways that surface weeks later in a finance report.&lt;/p&gt;

&lt;p&gt;At petabyte scale you cannot eyeball this. You need row counts, checksums, and — hardest of all — a deliberate inventory of every engine-specific type and function in the source, reviewed by a human who knows both engines, before conversion starts. The converter is a tool. The inventory is the work.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. The dual-write trap at cutover
&lt;/h2&gt;

&lt;p&gt;Dual-write feels safe: write to both systems, compare, switch over when they match. In practice, dual-write is where consistency goes to hide. Writes fail asymmetrically. Retries duplicate. One system applies an update the other rejected, and the divergence is invisible until reconciliation — if reconciliation is even looking at the right tables.&lt;/p&gt;

&lt;p&gt;And reconciliation at petabyte scale has its own trap: full-table validation costs real time and money, so teams sample. Sampling finds the errors you expected. It misses the one corrupted partition you didn't.&lt;/p&gt;

&lt;p&gt;The middle path that actually works: checksum validation on the tables that matter most (money, identity, state), row-count and aggregate validation everywhere else, and — this is the part teams skip — &lt;strong&gt;rollback criteria written down before cutover night.&lt;/strong&gt; At 3 a.m., nobody makes good decisions about whether to push forward or roll back. Decide at 3 p.m. the week before, in writing, with numbers.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Post-migration performance regressions
&lt;/h2&gt;

&lt;p&gt;The migration "succeeds." The data is there. And then the p99 latency graph starts climbing.&lt;/p&gt;

&lt;p&gt;Every long-lived database carries years of tuning debt in the best sense: query plans the old optimizer had settled into, statistics gathered over years, access patterns the application grew around. The new engine starts from zero. Its optimizer makes different choices — sometimes better, often just &lt;em&gt;different&lt;/em&gt; — and different loses to familiar when a checkout flow is timing out.&lt;/p&gt;

&lt;p&gt;The fix is unglamorous and systematic: capture the top queries by business impact (not by count), compare plans old versus new, rebuild statistics deliberately instead of waiting for auto-analyze, and work the list in impact order. I've seen this work recover 35% performance improvements over the pre-migration baseline once the tuning debt got repaid in the new engine's currency — but only for teams that budgeted the weeks to do it. If your migration plan ends at cutover, your plan ends too early.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. The cost surprises
&lt;/h2&gt;

&lt;p&gt;Finally, the bill. Migrations carry tail costs nobody modeled: replication traffic, cross-region transfer during months of parallel running, the old engine's license tail, and target storage replicated three ways because nobody revisited the defaults.&lt;/p&gt;

&lt;p&gt;The real savings come later, and they're structural. In my own cost-optimization work, the 40% reduction in managed database spend didn't come from the migration itself — it came from the rightsizing pass &lt;em&gt;after&lt;/em&gt; the dust settled: instances sized for the migration's peak rather than production's steady state, storage tiers revisited, reserved capacity bought once the new shape of the workload was known. Budget that pass into the plan, or the "savings" slide in the kickoff deck stays fiction.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pre-flight checklist
&lt;/h2&gt;

&lt;p&gt;Before any petabyte-scale migration, I want written answers to five questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;What is our measured bottleneck — source read, network, or target write — and what's the number?&lt;/li&gt;
&lt;li&gt;What is our expected CDC lag at cutover, at &lt;em&gt;peak&lt;/em&gt; change rate, and how did we rehearse it?&lt;/li&gt;
&lt;li&gt;Where is the human-reviewed inventory of engine-specific types and functions?&lt;/li&gt;
&lt;li&gt;What are the rollback criteria, in numbers, agreed before cutover night?&lt;/li&gt;
&lt;li&gt;What does the post-migration tuning and rightsizing pass look like, and who owns it?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Migrations don't fail because teams lack tools. They fail because scale turns small unknowns into schedule-ending surprises. The five places above are where those surprises live. Go look there first.&lt;/p&gt;

</description>
      <category>database</category>
      <category>aws</category>
      <category>googlecloud</category>
      <category>dataengineering</category>
    </item>
  </channel>
</rss>
