<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Shirley</title>
    <description>The latest articles on DEV Community by Shirley (@usernamevalid).</description>
    <link>https://dev.to/usernamevalid</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4123587%2F0e55382b-ca46-460a-8362-f5da2b64c088.jpeg</url>
      <title>DEV Community: Shirley</title>
      <link>https://dev.to/usernamevalid</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/usernamevalid"/>
    <language>en</language>
    <item>
      <title>The Day Your Monitoring Says Everything Is Fine While Your Users Are Losing Their Minds</title>
      <dc:creator>Shirley</dc:creator>
      <pubDate>Sun, 08 Nov 2026 18:40:01 +0000</pubDate>
      <link>https://dev.to/usernamevalid/the-day-your-monitoring-says-everything-is-fine-while-your-users-are-losing-their-minds-27ee</link>
      <guid>https://dev.to/usernamevalid/the-day-your-monitoring-says-everything-is-fine-while-your-users-are-losing-their-minds-27ee</guid>
      <description>&lt;p&gt;There's a particular kind of engineering incident that makes you question everything you thought you knew about production systems.&lt;/p&gt;

&lt;p&gt;Your dashboards are green. Your uptime monitor reports 99.99%. Your error rate is normal. CPU utilization looks healthy. Database connections are well within limits.&lt;/p&gt;

&lt;p&gt;And yet, your users are complaining that your application is broken.&lt;/p&gt;

&lt;p&gt;Pages take forever to load. Checkout randomly fails. Login works for some people but not others. Support tickets are piling up, and your engineering team is staring at a wall of green graphs wondering what everyone is talking about.&lt;/p&gt;

&lt;p&gt;Welcome to one of the most uncomfortable realities of software engineering: &lt;strong&gt;your system can be healthy according to your metrics and completely broken according to your users&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;I've been thinking about this because we spend so much time building monitoring systems that tell us whether our infrastructure is alive that we sometimes forget to ask whether our product is actually working.&lt;/p&gt;

&lt;p&gt;There's a difference.&lt;/p&gt;

&lt;p&gt;And that difference matters more than most teams realize.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Uptime is not the same thing as reliability&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;We have collectively become obsessed with uptime percentages.&lt;/p&gt;

&lt;p&gt;99.9%. 99.95%. 99.99%.&lt;/p&gt;

&lt;p&gt;These numbers look impressive in status reports and investor presentations. They make excellent dashboard decorations.&lt;/p&gt;

&lt;p&gt;But consider this scenario:&lt;/p&gt;

&lt;p&gt;Your application responds successfully to 99.9% of HTTP requests. Everything looks great.&lt;/p&gt;

&lt;p&gt;Except the 0.1% that fail are almost exclusively payment requests.&lt;/p&gt;

&lt;p&gt;Your homepage loads. Your product catalog works. Your API responds. Your health checks pass.&lt;/p&gt;

&lt;p&gt;But customers cannot complete purchases.&lt;/p&gt;

&lt;p&gt;Technically, your infrastructure is available. Practically, your business is losing money.&lt;/p&gt;

&lt;p&gt;This is why infrastructure-level monitoring alone is insufficient.&lt;/p&gt;

&lt;p&gt;Some practical takeaways:&lt;/p&gt;

&lt;p&gt;Monitor business-critical transactions, not just HTTP responses.&lt;br&gt;
Measure whether users can successfully complete important workflows.&lt;br&gt;
Track failed payments, abandoned checkouts, authentication failures, and other meaningful outcomes.&lt;br&gt;
Build alerts around customer impact rather than arbitrary infrastructure thresholds.&lt;br&gt;
A server returning HTTP 200 does not mean your application delivered value.&lt;/p&gt;

&lt;p&gt;It means a server returned HTTP 200.&lt;/p&gt;

&lt;p&gt;Those are very different statements.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Your average response time is hiding something&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Here's another dangerous habit: relying too heavily on averages.&lt;/p&gt;

&lt;p&gt;Imagine your API handles 100,000 requests.&lt;/p&gt;

&lt;p&gt;Most finish in 80 milliseconds. A small percentage take 8 seconds.&lt;/p&gt;

&lt;p&gt;Your average might still look perfectly reasonable.&lt;/p&gt;

&lt;p&gt;But those unlucky users are experiencing an application that feels broken.&lt;/p&gt;

&lt;p&gt;This is precisely why latency percentiles matter.&lt;/p&gt;

&lt;p&gt;P50 tells you what a typical request experiences. P95 and P99 tell you what happens to the people having the worst experience.&lt;/p&gt;

&lt;p&gt;And those people are often the ones who remember your product for all the wrong reasons.&lt;/p&gt;

&lt;p&gt;The same principle applies to background jobs, database queries, message queues, and third-party integrations.&lt;/p&gt;

&lt;p&gt;Averages are useful, but they can hide the exact problems that deserve your attention.&lt;/p&gt;

&lt;p&gt;If your team only looks at average latency, you're effectively ignoring the users who experience the worst version of your software.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;The dependency you don't control can become your biggest problem&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Modern applications are rarely isolated systems.&lt;/p&gt;

&lt;p&gt;Your application depends on payment providers, authentication services, email platforms, cloud infrastructure, analytics systems, package registries, and dozens of APIs maintained by other teams.&lt;/p&gt;

&lt;p&gt;You might have written excellent code.&lt;/p&gt;

&lt;p&gt;Your deployment might be flawless.&lt;/p&gt;

&lt;p&gt;Your infrastructure might be perfectly configured.&lt;/p&gt;

&lt;p&gt;Then a third-party service starts responding slowly, and suddenly your application is struggling.&lt;/p&gt;

&lt;p&gt;The interesting part is that the dependency doesn't necessarily need to go down.&lt;/p&gt;

&lt;p&gt;Sometimes it just gets slower.&lt;/p&gt;

&lt;p&gt;A payment API that normally responds in 200 milliseconds starts taking 4 seconds. Your application waits. Requests accumulate. Connection pools fill up. Threads become occupied. Eventually, unrelated parts of your application start suffering too.&lt;/p&gt;

&lt;p&gt;One slow dependency can become a much bigger incident than anyone expected.&lt;/p&gt;

&lt;p&gt;This is where engineering discipline matters:&lt;/p&gt;

&lt;p&gt;Set explicit timeouts for external requests.&lt;br&gt;
Use circuit breakers where appropriate.&lt;br&gt;
Implement retries carefully, with backoff and jitter.&lt;br&gt;
Make nonessential integrations fail gracefully.&lt;br&gt;
Avoid allowing one dependency to consume all your application's resources.&lt;br&gt;
Monitor third-party latency separately from your own application performance.&lt;br&gt;
Retries deserve special attention.&lt;/p&gt;

&lt;p&gt;A retry can be helpful when a temporary network problem occurs. But when a service is already struggling, thousands of clients retrying simultaneously can make the situation dramatically worse.&lt;/p&gt;

&lt;p&gt;Sometimes the most reliable thing your software can do is stop trying for a moment.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Observability is not a dashboard collection exercise&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;I've seen engineering teams with impressive monitoring setups.&lt;/p&gt;

&lt;p&gt;Dozens of dashboards. Hundreds of metrics. Logs flowing into centralized platforms. Distributed traces. Alerting rules for practically everything.&lt;/p&gt;

&lt;p&gt;And yet, when something breaks, nobody knows where to start.&lt;/p&gt;

&lt;p&gt;The problem isn't necessarily a lack of data.&lt;/p&gt;

&lt;p&gt;It's a lack of useful connections between the data.&lt;/p&gt;

&lt;p&gt;A log tells you something happened. A metric tells you how frequently something happened. A trace helps you understand how a request moved through your system.&lt;/p&gt;

&lt;p&gt;Together, they can tell a much more meaningful story.&lt;/p&gt;

&lt;p&gt;For example, imagine a customer reports that checkout failed.&lt;/p&gt;

&lt;p&gt;You want to be able to follow that transaction and understand:&lt;/p&gt;

&lt;p&gt;Which application instance handled the request?&lt;br&gt;
Which database queries were executed?&lt;br&gt;
Did the payment provider respond?&lt;br&gt;
Was there a timeout or retry?&lt;br&gt;
Did the request fail before or after the payment was authorized?&lt;br&gt;
Was the problem isolated to one deployment, region, or customer segment?&lt;br&gt;
Without that context, debugging becomes detective work.&lt;/p&gt;

&lt;p&gt;With it, you have a starting point.&lt;/p&gt;

&lt;p&gt;And there's another important detail: observability needs to exist before the incident.&lt;/p&gt;

&lt;p&gt;Adding useful instrumentation after production breaks is a little like installing a security camera after someone has already stolen your equipment.&lt;/p&gt;

&lt;p&gt;You might eventually figure out what happened, but you've lost the opportunity to see it clearly.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Alert fatigue is an engineering problem, not a notification problem&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;There's a dangerous pattern in many engineering organizations.&lt;/p&gt;

&lt;p&gt;An alert fires. Someone investigates. It turns out to be nothing serious.&lt;/p&gt;

&lt;p&gt;Another alert fires. Same story.&lt;/p&gt;

&lt;p&gt;Then another.&lt;/p&gt;

&lt;p&gt;Eventually, engineers stop treating alerts as urgent signals and start treating them as background noise.&lt;/p&gt;

&lt;p&gt;This is alert fatigue.&lt;/p&gt;

&lt;p&gt;And it's not harmless.&lt;/p&gt;

&lt;p&gt;When every minor deviation triggers a notification, genuinely important incidents become harder to recognize.&lt;/p&gt;

&lt;p&gt;The solution isn't necessarily more sophisticated alerting software.&lt;/p&gt;

&lt;p&gt;It's better alert design.&lt;/p&gt;

&lt;p&gt;A good alert should answer three questions:&lt;/p&gt;

&lt;p&gt;Is something actually wrong?&lt;br&gt;
Does someone need to take action?&lt;br&gt;
What is the likely customer impact?&lt;br&gt;
If an alert doesn't lead to an investigation or a meaningful action, it probably needs to be reconsidered.&lt;/p&gt;

&lt;p&gt;I would rather have five alerts that engineers trust than 500 alerts that everyone has learned to ignore.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;The best incident response starts before the incident&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;There's a misconception that incident response is something you figure out when production goes down.&lt;/p&gt;

&lt;p&gt;It isn't.&lt;/p&gt;

&lt;p&gt;By that point, people are stressed, customers are frustrated, and every minute feels expensive.&lt;/p&gt;

&lt;p&gt;The teams that handle incidents well have already established some basic habits:&lt;/p&gt;

&lt;p&gt;They know who owns critical services.&lt;br&gt;
They have documented rollback procedures.&lt;br&gt;
They understand their dependency relationships.&lt;br&gt;
They practice restoring backups.&lt;br&gt;
They have clear escalation paths.&lt;br&gt;
They know how to communicate with customers.&lt;br&gt;
They conduct blameless postmortems after meaningful incidents.&lt;br&gt;
One practice I particularly like is running failure exercises.&lt;/p&gt;

&lt;p&gt;Intentionally introduce a controlled failure in a non-production environment. Simulate a database slowdown. Make a dependency unavailable. Test what happens when a queue falls behind.&lt;/p&gt;

&lt;p&gt;Then ask a simple question:&lt;/p&gt;

&lt;p&gt;Would our monitoring tell us something meaningful is wrong before customers start reporting it?&lt;/p&gt;

&lt;p&gt;If the answer is no, you've found something worth fixing.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;There's a business lesson hiding inside all of this&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Engineering teams often measure reliability from the perspective of the system.&lt;/p&gt;

&lt;p&gt;Businesses experience reliability from the perspective of the customer.&lt;/p&gt;

&lt;p&gt;Those perspectives don't always align.&lt;/p&gt;

&lt;p&gt;A database can be available while queries are painfully slow. An API can return successful responses while delivering incorrect data. A deployment can complete without errors while breaking an important customer workflow.&lt;/p&gt;

&lt;p&gt;The technical definition of success is not always the customer's definition of success.&lt;/p&gt;

&lt;p&gt;And that's probably the most important lesson here.&lt;/p&gt;

&lt;p&gt;We shouldn't build monitoring systems simply to prove that our infrastructure is running.&lt;/p&gt;

&lt;p&gt;We should build them to understand whether our software is doing what people depend on it to do.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;The takeaway&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Every production system eventually encounters something its engineers didn't anticipate.&lt;/p&gt;

&lt;p&gt;A dependency becomes slow. A database behaves unexpectedly. A deployment introduces an edge case. A seemingly harmless traffic spike exposes a bottleneck nobody noticed during testing.&lt;/p&gt;

&lt;p&gt;That's normal.&lt;/p&gt;

&lt;p&gt;What's less acceptable is discovering that your monitoring system had no idea any of it was happening.&lt;/p&gt;

&lt;p&gt;Green dashboards are comforting, but comfort isn't the same thing as confidence.&lt;/p&gt;

&lt;p&gt;Build systems that measure real customer outcomes. Pay attention to the slowest requests, not just the averages. Instrument critical workflows. Practice failure scenarios. And make sure your alerts tell people when something genuinely needs attention.&lt;/p&gt;

&lt;p&gt;Because the most dangerous production incident isn't necessarily the one that brings your servers down.&lt;/p&gt;

&lt;p&gt;Sometimes it's the one where everything looks healthy while your customers quietly give up trying to use your product.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;strong&gt;Have you ever experienced an incident where every dashboard was green but the application was effectively broken? What eventually helped you find the problem? Share your story in the comments.&lt;/strong&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Sources&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Google Cloud, Site Reliability Engineering (SRE) Books — Practical guidance on reliability engineering, monitoring, incident response, and service-level objectives.&lt;/p&gt;

&lt;p&gt;Google Cloud, Monitoring Distributed Systems — Why monitoring should focus on meaningful signals, including latency, traffic, errors, and saturation.&lt;/p&gt;

&lt;p&gt;Google Cloud, The Four Golden Signals — A framework for understanding latency, traffic, errors, and saturation.&lt;/p&gt;

&lt;p&gt;OpenTelemetry, Documentation — An introduction to observability through metrics, logs, and distributed traces.&lt;/p&gt;

&lt;p&gt;AWS, Timeouts, retries, and backoff with jitter — Why poorly designed retries can amplify failures in distributed systems.&lt;/p&gt;

&lt;p&gt;Google Cloud, Site Reliability Engineering: Incident Management — Best practices for coordinating and responding to production incidents.&lt;/p&gt;

&lt;p&gt;Atlassian, Incident Management — Guidance on incident response, communication, and post-incident learning.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;These resources provide additional technical context for the reliability and observability practices discussed in this article.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>softwareengineering</category>
      <category>testing</category>
      <category>discuss</category>
    </item>
    <item>
      <title>What Apple’s “Crackgate” Teaches Software Teams About Rare Failures published: false</title>
      <dc:creator>Shirley</dc:creator>
      <pubDate>Tue, 27 Oct 2026 19:59:35 +0000</pubDate>
      <link>https://dev.to/usernamevalid/what-apples-crackgate-teaches-software-teams-about-rare-failurespublished-false-2lfi</link>
      <guid>https://dev.to/usernamevalid/what-apples-crackgate-teaches-software-teams-about-rare-failurespublished-false-2lfi</guid>
      <description>&lt;p&gt;Apple’s first foldable iPhone, the iPhone Duo, has barely hit the market, and users are already posting photos of screens with the same horizontal crack. The internet named it “Crackgate” within days, echoing “Antennagate” from 2010. Apple has acknowledged that a small number of units have inner display problems and is routing affected users to AppleCare for replacements.&lt;/p&gt;

&lt;p&gt;I don’t build phones. But reading about it, I kept thinking that this is a story about shipping software too. The failure modes are the same, and so are the lessons.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Your test suite can’t see what only happens one time in a thousand&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A former Apple screen engineer quoted in the coverage made a point every developer should tattoo somewhere visible. When a defect is rare enough, you can run a thousand units through testing and never see it. You need at least one failure to show up before you can start chasing it.&lt;/p&gt;

&lt;p&gt;That’s the flaky-test problem and the race-condition problem, scaled up. A bug that appears in 0.1% of runs will pass your CI pipeline nearly every time. It will also fail for someone in production within hours, because production is where the volume is.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Some practical takeaways:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Treat “it passed” as evidence, not proof&lt;/strong&gt;. A green build means you didn’t observe a failure, not that none exists.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Invest in observability before you need it.&lt;/strong&gt; If a rare failure appears, you want the logs, traces, and device or environment data already waiting.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Take rare reports seriously.&lt;/strong&gt; One user’s weird screenshot can be the first visible tip of a systemic problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pattern recognition is a superpower&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One analyst noted that the consistency of the reports, the same crack in roughly the same place, suggests something systematic, perhaps tolerances in the hinge or how the display is supported during folding. Random damage looks random. A repeated pattern points to a cause.&lt;/p&gt;

&lt;p&gt;This is exactly how good debugging works. When dozens of bug reports look alike, resist the urge to treat them as dozens of separate problems. Look for the common factor: the same OS version, the same region, the same batch of dependencies, the same deploy window. The article also raises the possibility of a defective batch of components from a supplier, which is the hardware version of a bad dependency release.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rollbacks are easy in software, which is exactly why we skip the discipline&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Hardware teams live with a brutal constraint: you can’t push a patch to a cracked screen. That’s why Apple’s options here are replacements, service programs, and possibly something bigger. A recall would be unusual for the company, but it’s on the spectrum.&lt;/p&gt;

&lt;p&gt;Software teams have the opposite problem. We can fix almost anything after the fact, so we get sloppy about before the fact. Feature flags, canary deploys, and staged rollouts exist because the ability to fix things quickly is not a substitute for limiting blast radius. The article mentions that earlier this year Apple shipped a software fix for a cellular connectivity problem affecting some customers on one carrier. Fast fixes are great, but the people who hit the bug first still had the bad experience.&lt;/p&gt;

&lt;p&gt;If you can roll out to 1% before 100%, do it. The 1% will find your thousand-to-one bug for you.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The communication lesson is older than the bug&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The 2010 episode is remembered less for the antenna than for the response. Steve Jobs initially brushed off complaints by implying users were holding the phone the wrong way, and Apple eventually offered free protective bumpers.&lt;/p&gt;

&lt;p&gt;Compare that with the current statement, which acknowledges the issue plainly and tells people what to do. Whatever you think of the company, the second approach is the one engineering teams should copy in an incident:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Acknowledge quickly.&lt;/strong&gt; Silence reads as denial.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Say what you know and don’t know&lt;/strong&gt;. Apple admitted a problem without claiming to know its scale.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Give users a clear next step.&lt;/strong&gt; “Contact support for a replacement” beats “we’re looking into it.”&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Never blame the user for using the product as designed.&lt;/strong&gt; If folding a foldable breaks it, that’s a design problem.&lt;/p&gt;

&lt;p&gt;I’d add one for status pages and postmortems: don’t let a competitor tell your story first. Samsung ran an ad within days of the news. Your equivalent might be a rival’s blog post about your outage.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;High expectations make small failures loud&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The Duo costs upwards of two thousand dollars, and the company had publicly emphasized waiting until the engineering was right instead of shipping something half-baked. That’s a strong promise, and a strong promise raises the stakes on every defect.&lt;/p&gt;

&lt;p&gt;Software has its own version of this. The more you market reliability, polish, or “it just works,” the less forgiveness you get for a visible failure. A scrappy side project can have bugs and be charming. A payments platform that has built its brand on trust cannot.&lt;/p&gt;

&lt;p&gt;That doesn’t mean promise less. It means match your testing, monitoring, and incident readiness to the promises you make.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The takeaway&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Even a company with enormous engineering resources and tightly controlled supply chains can ship a rare, hard-to-catch defect. If it can happen to them, it can happen to your team. The question isn’t whether a one-in-a-thousand bug will reach users. It’s whether you’ll find out fast, understand the pattern, limit the damage, and tell people the truth.&lt;/p&gt;

&lt;p&gt;Build for that moment before it arrives.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;strong&gt;Have you ever shipped a bug that your tests had no realistic way to catch? How did you find it, and what changed afterward? Share it in the comments.&lt;/strong&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sources:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Michael Acton, “Apple hit by ‘Crackgate’ outcry over iPhone Duo screen problems,” Financial Times, San Francisco. &lt;a href="https://www.ft.com/content/324875fb-8631-4339-8516-6057de9ac085" rel="noopener noreferrer"&gt;https://www.ft.com/content/324875fb-8631-4339-8516-6057de9ac085&lt;/a&gt; (add the publication date from the article page; FT content may require a subscription)&lt;/p&gt;

</description>
      <category>testing</category>
      <category>devops</category>
      <category>softwareengineering</category>
      <category>discuss</category>
    </item>
  </channel>
</rss>
