Do you have guards? or do you work at night?

twitter logo ・1 min read

Yesterday i did office hours as usual and then at 3am i did a deploy, for that thing of the life, the deploy was a failure and i worked until 6am. So, my question is, is this normal? do you work at night? do you do deploys at night? or how do you manage it?

Second, this night i entry on my team guard, 24x7 paying attention to opsgenie* and i don't receive any monetary retribution for this. I'm curios about how your company compensate the guards if you have it and how hard are it.

*it is a 7 days, 24 hours passive guard, if the cellphone call me, it doesn't matter the time, I have to work until the problem is resolved.

twitter logo DISCUSS (15)
markdown guide
 

This sounds horrific. Why are you deploying at 3am if it's possible to fail - what's the mission critical item in the deployment?

Guards - well, this is called an 'on call' duty from where I've been. There was a small monetary compensation from going on the rota. The trick is to make sure that your code can't break.

This code that's keeping you awake - did you write it? Did you test it? Is the deployment automated? Does the automated deployment run automated tests? It's stuff like that that let's me sleep at night.

 

we are deploying at 3am because we have less traffic at that time.

my teams apps doesn't crash regularly, but, external services or other teams APIs does, so, i usually have to contact the other team and ask what is happening and get some info about the issue, the same to external service.

I work in a +1500 devs company, we follow standards, ci/cd, but, shit happens πŸ’©

 

Blue/Green, rolling, canary - these are deployment strategies you want to look into.

If you have a CI/CD pipeline, and it is being used correctly, there is only very few reasons to not have an constant deployment process.

 

For a sys admin/devops/SRE is normal to have oncalls shifts, but to deploy at night I do not think so.

I suggest reading some devops or SRE books to see how the tech companies handle these cases.

As a summary, no, you do not deploy when you are alone, maybe a rollback. You touch the production if you have your back assured, as in other teammates to help you. You mitigate the "failure radius" and less on the "why it happened".

we are deploying at 3am because we have less traffic at that time.

I guess you are not using a deployment strategy, with canary, soft rollouts and blue/green. The traffic is irrelevant, yesterday I saw a Panel on a team with 200.000 requests per seconds doing deploys mid day, is actually very important to do them during the office hours when you have all hands on deck to help investigate and speed things up.

 

I'm a developer, we have a SRE team too, and we have these books at the office :D

 
 

Late nights happen. Either we schedule maintenance outside business hours (insofar as is possible) or maybe something goes wrong and triggers an alert; in the latter case, I may not be explicitly on call but I'll jump in if I catch it. I took the job knowing this would happen and factored it into my consideration of their offer. We're a small startup and aren't super formal so if something keeps me up late I'll usually sleep in the following morning to balance things out.

 

That sounds fear and i thinks helping occasionally it's okey.

 

1) No, yes, kinda. Depends. Any project I am on I advocate for the ability to deploy whenever, triggered by whoever. Specifically: master branch is always deployable. If I had to work 3a-6a, you bet your a** I will not be in the office the following day, period.

As a DevOps Engineer most places expect an 'on-call' rotation. This means top level SLA system issues have to be acknowledged within 30 min. Just means 'I have received the alert, will get back to you when it is fixed'. The time it takes to resolve comes off the next days office hours.

 

I agree with you, i'm describing the situation and i want to see yours! I'm working from home today, at least my boss is more humanistic oriented than the rest, so he allows me to work from home and sleep after a ruff night.

 

If you're a salaried employee then in most places you won't be eligible for overtime. In some cases companies will offer "in lieu" time, where if you work late one day you can bank that time and take it off some other day, but that's entirely driven by company policy.

You'll want to be sure that your obligations are documented in some capacity so that you're certain of what they are, plus should any dispute arise in the future you have records to back up your claims.

Small companies tend to put a lot of pressure on their developers, so it's sort of understood to come with the territory. Normally that means being on-call, where if a service monitoring alert goes off you have to react. It doesn't normally mean watching dashboards at all hours.

It sounds like your company needs a lot more devops automation to ensure people aren't stuck watching dashboards all the time.

 

Echoing what a lot of people have said here - on call rotations, or guards, are perfectly normal and expected. But anything waking you up at 3am should be mission critical - like rebooting a service that OOM'd or something. Regular deploys and debugging should not be part of your overnight duties.

A 7 day rotation with 24 hours of being a secondary on call person sounds like a reasonable shift. But you should talk with your upper management team about reducing overnight calls and alerts as a key performance metric of your team. I can't imagine that you're performing your best during regular business hours if you wake up at 3 am and debug until 6.

If nothing else, start tracking overall call volume for your team. If you're not already using a service like PagerDuty to manage alerts, start doing that. It makes scheduling shifts and tracking metrics much easier. Try to identify high call volume times of day and/or days of week. Figure out if there's a common pattern causing them. Bring it up as part of your planning process that this is tech debt you need to pay down for your team to be more efficient.

Sorry if I'm being overly prescriptive here, but the schedule you're working sounds extremely unpleasant. I want better for you!

 

Thanks! i can see the state of on call rotations more clearly, at least here in southamerica is tough.

 

Yikes, that sounds really rough!

I know it's different for SREs and so on, and most of our systems aren't services that need full uptime, but at my office we're pretty much all out by 5:30pm. When there's an emergency or a crunch, we're usually pretty open to folks working from home to cut transit time or be able to do something at an awkward time of day. The team lead is really adamant that we should be able to clock out, though, and that we shouldn't be working hours we're not getting paid for, which I think is really nice.

When we've done "night" deploys we've usually done them more in the evening and check with any stakeholders/clients about when the peak hours are so we avoid them. We get our 24/7 coverage from actually having a team in an opposite time zoneβ€”we have a small team in Ukraine that oversees our nighttime coverage since it's daytime for them and they can respond faster if a site goes down or there's a catastrophic bug that needs fixing.

 
Classic DEV Post from Jan 16

What's your typical process for reviewing a pull request in GitHub?

I'm never sure whether I'm using this tool as effectively and efficiently as po...

Jacinto Jaimez profile image
https://www.linkedin.com/in/jacintojaimez/

πŸ‘‹ Hey dev.to reader.

Do you prefer sans serif over serif?

You can change your font preferences in the "misc" section of your settings. ❀️