Sergey Shinder - DEV Community

Sergey Shinder

Nov 2

Our label service ran out of local ports at four hundred and seventy requests a second

#sergeyshinder #sre #linux #networking

2 min read

Sergey Shinder

Nov 2

Helm reported a successful release while its data job was still failing in the first region

#sergeyshinder #helm #kubernetes #releaseengineering

2 min read

Sergey Shinder

Nov 2

One apostrophe in a delivery address ended our despatch advice in the middle of a segment

#sergeyshinder #integration #edi #api

2 min read

Sergey Shinder

Nov 2

Our disk alert waited for ninety seven percent and Postgres stopped at ninety five

#sergeyshinder #monitoring #linux #postgres

1

2 min read

Sergey Shinder

Nov 2

Our third search replica waited six hours for a node in a zone our node group did not cover

#sergeyshinder #kubernetes #storage #aws

2 min read

Sergey Shinder

Nov 2

We changed the user data on our integration server and Terraform restarted it without running a line

#sergeyshinder #terraform #aws #iac

2 min read

Sergey Shinder

Nov 2

Our deploy workflow waited for CI to finish and never asked whether it had passed

#sergeyshinder #cicd #githubactions #devops

2 min read

Sergey Shinder

Nov 1

Our Redis replica spent three days restarting the same sync

#sergeyshinder #redis #sre #reliability

1

2 min read

Sergey Shinder

Nov 1

We moved one Go variable and every copy of our new CLI was told to upgrade

#sergeyshinder #go #releaseengineering #automation

1

2 min read

Sergey Shinder

Nov 1

The schema registry approved a change that only worked if we deployed first

#sergeyshinder #kafka #integration #avro

2 min read

Sergey Shinder

Nov 1

One slow count query took every metric our dispatch service had off the dashboard

#sergeyshinder #prometheus #observability #monitoring

2 min read

Sergey Shinder

Nov 1

Our statement job ran out of memory at seventy percent and Kubernetes started it again from the first customer

#sergeyshinder #kubernetes #batch #containers

2 min read

Sergey Shinder

Nov 1

We passed null to say use the default and Terraform switched off deletion protection

#sergeyshinder #terraform #aws #iac

2 min read

Sergey Shinder

Nov 1

One failing job in our nightly matrix cancelled the eleven that would have told us more

#sergeyshinder #cicd #githubactions #testing

2 min read

Sergey Shinder

Oct 30

Our notification service ran out of memory at two in the morning and stayed half alive until nine

#sergeyshinder #sre #java #kubernetes

1

2 min read

Sergey Shinder

Oct 30

Every deploy broke our portal for anyone who already had it open

#sergeyshinder #releaseengineering #frontend #aws

1

2 min read

Sergey Shinder

Oct 30

Our invoice links were valid for seven days and stopped working after an hour

#sergeyshinder #aws #integration #s3

2 min read

Sergey Shinder

Oct 30

Every info line our Python services wrote arrived in Cloud Logging as an error

#sergeyshinder #observability #python #gcp

2 min read

Sergey Shinder

Oct 30

We added a sleep before shutdown and our pods ran out of time to shut down

#sergeyshinder #kubernetes #containers #kafka

2 min read

Sergey Shinder

Oct 30

Ansible changed our nginx config on forty servers and reloaded it on twenty nine

#sergeyshinder #ansible #nginx #iac

1

2 min read

Sergey Shinder

Oct 30

Our migrations passed in CI as a superuser and failed in the first new database we built

#sergeyshinder #cicd #postgres #testing

2 min read

Sergey Shinder

Oct 30

CloudWatch had a slow hour and our delivery slot service stopped because it could not finish writing a log line

#sergeyshinder #sre #docker #logging

1

2 min read

Sergey Shinder

Oct 30

Helm upgraded our operator and kept its old CRD, and the cluster threw the new setting away

#sergeyshinder #releaseengineering #kubernetes #helm

2 min read

Sergey Shinder

Oct 30

Our calls to the carrier failed for twenty minutes after every deploy and then fixed themselves

#sergeyshinder #grpc #integration #networking

2 min read

Sergey Shinder

Oct 28

We put the error rate in our alert's labels and it never stayed the same alert long enough to fire

#sergeyshinder #observability #prometheus #alerting

2 min read

Sergey Shinder

Oct 28

Our document store took forty one minutes to start because Kubernetes rechecked every file on its disk

#sergeyshinder #kubernetes #storage #containers

2 min read

Sergey Shinder

Oct 28

One secret in our task definition hid the database change sitting next to it

#sergeyshinder #terraform #aws #iac

2 min read

Sergey Shinder

Oct 28

A hundred and eighty six tests passed on every laptop and CI never collected one of them

#sergeyshinder #cicd #python #testing

2 min read

Sergey Shinder

Oct 28

One queue nobody was reading filled the disk and RabbitMQ stopped every publisher we had

#sergeyshinder #sre #rabbitmq #messaging

2 min read

Sergey Shinder

Oct 28

Our rollout finished in two minutes and the new pods started crashing in the fifth

#sergeyshinder #releaseengineering #kubernetes #node

2 min read

Sergey Shinder

Oct 27

We streamed our claims file to save memory and the insurer's gateway asked how long it was

#sergeyshinder #api #integration #python

1

2 min read

Sergey Shinder

Oct 27

We stretched one Alertmanager timer to quiet a disk warning and our payment alert waited behind it

#sergeyshinder #observability #prometheus #alerting

1

2 min read

Sergey Shinder

Oct 27

Our ingress saw every visitor as one of our own nodes and let them reach the admin pages

#sergeyshinder #kubernetes #networking #security

1

2 min read

Sergey Shinder

Oct 27

Ansible turned off password logins on every server and sshd read another file first

#sergeyshinder #ansible #linux #security

1

2 min read

Sergey Shinder

Oct 27

Our nightly image rebuild stopped because nobody had pushed to the repository for sixty days

#sergeyshinder #cicd #githubactions #docker

1

2 min read

Sergey Shinder

Oct 19

A cleanup job that listed every cart key paused Redis for the whole site

#sergeyshinder #sre #redis #performance

1

2 min read

Sergey Shinder

Oct 19

Our signing certificate expired and took every installer we had ever shipped with it

#sergeyshinder #releaseengineering #windows #security

1

2 min read

Sergey Shinder

Oct 19

Our new API gateway lowercased every header and no partner could place an order

#sergeyshinder #api #integration #aws

1

2 min read

Sergey Shinder

Oct 19

Our backup dashboard stayed green for five weeks after the job had moved

#sergeyshinder #observability #prometheus #backups

1

2 min read

Sergey Shinder

Oct 19

Our log shipper moved to the host network and could no longer find the cluster

#sergeyshinder #kubernetes #dns #logging

2

2 min read

Sergey Shinder

Oct 19

We recreated our CI role with the same name and lost control of our encryption keys

#sergeyshinder #terraform #aws #security

2

2 min read

Sergey Shinder

Oct 19

Our deploy form had an unticked box and every deploy rebuilt the search index

#sergeyshinder #cicd #githubactions #automation

2

2 min read

Sergey Shinder

Oct 19

One export job switched off the query timeout for strangers

#sergeyshinder #sre #postgres #database

1

2 min read

Sergey Shinder

Oct 16

Our routing table outgrew kubectl apply and half a release went out

#sergeyshinder #releaseengineering #kubernetes #kubectl

2 min read

Sergey Shinder

Oct 16

Every supplier file with a space in its name was refused as access denied

#sergeyshinder #api #integration #aws

2 min read

Sergey Shinder

Oct 16

Kubernetes knew why checkout failed at three in the morning and forgot by four

#sergeyshinder #observability #kubernetes #monitoring

1

2 min read

Sergey Shinder

Oct 16

Our autoscaler stopped scaling the day we added a sidecar without a CPU request

#sergeyshinder #kubernetes #autoscaling #containers

2 min read

Sergey Shinder

Oct 16

Terraform said no changes and our Lambda kept running last month's code

#sergeyshinder #terraform #aws #serverless

2 min read

Sergey Shinder

Oct 16

Our build cache was saved once in January and never again

#sergeyshinder #cicd #githubactions #maven

2 min read

Sergey Shinder

Oct 15

Our standby received every change for five hours and applied none of them

#sergeyshinder #sre #postgres #database

1

2 min read

Sergey Shinder

Oct 15

A deploy killed halfway left Helm refusing every release after it

#sergeyshinder #releaseengineering #helm #kubernetes

1

2 min read

Sergey Shinder

Oct 15

Our sync sent null for every field it had not loaded and the partner deleted them

#sergeyshinder #api #integration #json

1

2 min read

Sergey Shinder

Oct 15

Every log line over sixteen kilobytes arrived in pieces and none of them counted as an error

#sergeyshinder #observability #logging #kubernetes

1

2 min read

Sergey Shinder

Oct 15

Our delivery photos stopped uploading the week we moved to ingress-nginx

#sergeyshinder #kubernetes #nginx #containers

1

2 min read

Sergey Shinder

Oct 15

One depends_on line taught us to stop reading our Terraform plans

#sergeyshinder #terraform #iac #aws

1

2 min read

Sergey Shinder

Oct 15

Our integration tests skipped themselves on every Dependabot pull request

#sergeyshinder #cicd #githubactions #testing

1

2 min read

Sergey Shinder

Oct 14

Our quote service stopped at lunchtime with its CPU at two percent

#sergeyshinder #sre #java #concurrency

2

1

2 min read

Sergey Shinder

Oct 14

We rolled back the image and kept the configuration that broke it

#sergeyshinder #releaseengineering #kubernetes #rollback

1

2 min read

Sergey Shinder

Oct 14

Our price job put back the stock our stock job had just sold out

#sergeyshinder #api #integration #http

1

2 min read

Sergey Shinder

Oct 14

Our dashboard counted every failed refund except the first one on each pod

#sergeyshinder #observability #prometheus #metrics

1

2 min read

Writing Debut