DEV Community: Molly Struve (she/her)

Interview Prep Questions

Molly Struve (she/her) — Wed, 18 Aug 2021 20:12:12 +0000

Followup post for Setting Yourself Up for Interview Success

Below is a list of questions that I found extremely helpful when preparing for my interviews. Most of them came directly from the interviews I had and some were general ones I found online.

When I used these to prep I put them on flash cards. I wrote the question on one side and my answer (or answers) on the other. Then I would go through them at the beginning of each week I was interviewing to prepare for my interviews.

For reference, I was interviewing for Senior, Staff, and Engineering Manager positions with a focus on SRE (Site Reliability Engineering) and Reliability roles. I broke up the questions into categories to make them easier to sort through. They are in no particular order. I only numbered them to make them easy to reference. Using my answers to these questions I was able to answer every question that was thrown at me during my interviews.

General Questions

What do you want out of your next role?
Tell me about a time you disagreed with someone. How did you handle it? What was the outcome?
- The follow up questions might not get asked but get in the habit of answering those even when giving the initial answer.
What are your greatest strengths?
What are your greatest weaknesses?
- You likely will not directly get asked the above questions, but if you are aware of what your strengths are then you can work them into your answers for other questions. If you know what your weaknesses are you can also mention those and how you are working to improve them. The key is to not have any blindspots.
What is a time when something took a lot out of you and what did you learn?
List the major achievements you are most proud of at your current job.
Tell me about someone you look up to and why?
Tell me about a time you made a mistake.
What was a piece of feedback that was hard to hear but helped you improve?

Sr./Staff Level IC Questions

Tell me about a complex system you designed.
Tell me about a large project you had to get buy in for. What was it and how did you get the buy in you needed?
Tell me about a time you failed and what you learned from it.
- Even if they don't ask for what you learned you always include that. It shows that you are humble and have the ability to grow.
What do you value highly when working with software? (technical values)
- Example answer: "I highly value simplicity bc I have found it leads to more reliable software. I also value automation bc humans doing things manually usually increases the chances of error." Expand more on your answer, but that gives you an idea of the kinds of answers you would give.
Tell me about a time you had to make a decision that involved short-term sacrifices for long-term gains.
Tell me about a time when you solved a complex problem and how you went about it.
- Note this is different than #1 which asks about designing a complex system. I choose a story about a gnarly bug I had to fix.
What is the most challenging part of being in an engineering leader for you?

SRE/Reliability IC Questions

Tell me about an incident that you ran?
How do you define observability? How have you improved observability on past systems you have worked with?
What tools have you built that were able to aid developers and/or fellow SREs?
Give me an example of a time you made something more efficient.
Give me an example of a time you improved a process. (could be team process or technical process)
- I talked about improving an on-call rotation for these types of questions.
What does reliability mean to you?
What is your preferred testing strategy?
- Here you want to talk high level like unit vs end-to-end, smoke tests, canary deployments, etc. Give examples of what has worked well or not for you in the past.
Give me an example of a calculated risk that you took where speed was critical.
What is your definition of SRE?

Manager Questions

What is your management style?
Tell me about the person you are most proud of empowering and growing.
Tell me about a time when you had to give someone hard feedback.
How do you deal with competing priorities?
- When you get "how do you..." questions, answering with your strategy is good, but having a quick story to follow it up as an example is even better.
How do you gain trust from your team?
What do you look for when you are hiring?
How do you determine if someone is technically competent for a job?
How do you keep the jerks out?
Have you ever had a large project killed?
How do you broach the topic of technical debt when having to explain it to leadership and other stakeholders?
What does the diversity of your current team look like?
Tell me about a time you missed a deadline and how you handled it.
When was a time you failed as a manager and how did you handle it and make it right?

Depending on your experience, you may not have an answer for all of the "Tell me about a time..." questions. That is OK! Be truthful about it and then explain how you would handle the situation if you found yourself faced with it. For example, I never have had a large project killed but I talked about how I would be objective and make sure my team fully understood why it had happened.

If you have any other questions you would like to add to this list, feel free to drop them in the comments below 👇

Setting Yourself Up for Interview Success

Molly Struve (she/her) — Wed, 18 Aug 2021 20:11:53 +0000

When it comes to interviewing, everyone has their own style and way they like to approach it. When I started looking for my next opportunity, I was not quite sure what to expect. I read a few blog posts to prep myself. These were incredibly helpful, but I found a few things even beyond their recommendations that helped me to succeed and that is what I want to share with you all. My approach may not be ideal for everyone but it worked very well for myself and using it I was able to land five job offers.

Interview Often

Many folks will tell you that interviewing is a skill and you should make sure to get in some practice rounds before aiming towards a company you really want. I 100% found this to be the case. The first couple of phone screens I did I had to pause and think about the answers to the questions. As things went along, I was asked similar questions over and over and it got much easier.

However, even with a couple of practice rounds, I still felt a bit stressed and anxious going into interviews. Then, as my interviewing frequency picked up, I noticed my anxiety nearly went away. When I had an interview scheduled almost every day my anxiety was almost non-existent because it felt like just one more meeting on my schedule.

Once I had this realization I started to schedule interviews on a consistent cadence each week. This not only lowered my anxiety, but did wonders for my ability to solve the technical problems. Instead of getting nervous before a technical session, I would get focused. Interviewing on a consistent basis got me into a nice groove when it came to tackling technical interviews such as coding or systems design problems.

I even went so far as to schedule a set of interviews with a company solely for the reason of getting me in the groove and prepped for one of the final companies I interviewed with. Not only did that extra company help prep me, but it turned out to be a fabulous interview and I got an offer that I seriously considered in the end.

Flash Cards

The first couple of roles I did hiring screens for determined I was not a great fit. One of the roles I actually thought I was a great fit for but realized I had not presented my skills as well as I could have during the phone screen.

Immediately after those phone screens I decided it was time to get organized and take interviewing more seriously. I sat down and wrote out the questions they asked that, at the time, I didn't think I gave good answers to. Then I took time to come up with much better answers. I combed through my mind to find examples that would better highlight my skillset. In addition to writing down answers to the questions I was asked in the phone screens, I collected some more generic technical interview questions from various websites and wrote down answers for those as well.

I initially put all of this into a Google doc. The Google doc was great for getting my thoughts written down, but I wanted something that I could use repeatedly to prep. To achieve this I went back to my college days and used the tried and true study method of flash cards! I made flash cards with the questions on one side and my answers on the other. Then, like in school, I would run through the flash cards at the beginning of each week to make sure I had a quick recall of all the stories and highlights of my career.

Even though the interview questions didn't always exactly match those on my flash cards, the flash cards helped remind me about the key points throughout my career. Those key points and experiences were easily transferable to many types of questions. For folks that are interested, I wrote a second post listing out all of the questions I used to prepare.

Interview Prep Questions

Molly Struve (she/her) ・ Aug 18 '21

#interviewing #job #career

Professional Interview Prep

Before starting this process I looked into companies and websites that I could use for interview prep. I felt pretty confident in my coding ability, but wanted to learn more about the systems design portion that many interviews have. To fill this void I signed up for Exponent. Exponent is a bit pricey, but it was completely worth it for me. One nice aspect is that they have some free content available so you can see if it is right for you and what you want to accomplish. I only ended up subscribing to it for two months.

I found their guide to SWE interviewing to have a lot of great information in it. One part that was crucial in helping me was the systems design section. I had never done a systems design interview before so I had no clue about how to approach one.

Exponent had many examples of folks answering system design questions which gave me an idea of what a good answer included. They also had lots of literature about how to break down the system design questions and what to cover when answering them. Once again, I ended up making flash cards around system design question elements and ran through those periodically to keep myself sharp on that front.

For more details on how to I tackled my system design interviews stay tuned for a follow up blog post. 😎

Take Notes

During your interviews take notes! Especially if you are interviewing at multiple companies, it can be very hard to keep them all straight. I have Google docs for each company I interviewed with that have extensive notes from each process.

These docs contain everything from the questions I asked, to the background of each interviewer, to any feelings I felt during the interview. These notes were invaluable at the end of the process when it came to decision time.

Give yourself 30 min before each interview

Scheduling interviews around work can be challenging, but one thing I found is that scheduling in downtime right before is crucial to get yourself in the mental state to interview. My routine involved the following:

Gather up my notes already compiled for the company and write down questions I wanted to ask
Research the interviewers if I had their names
Turn off all notifications and close Slack
Open any necessary tools (code editor, whiteboard tool, etc)
Go to the bathroom and grab some water
Power pose for a min and then have a quick dance party to get rid of any last min nerves and pump myself up.

That last point is not for everyone, but for me, listening to some pump up music while dancing makes me feel really good about myself and gets me in a good headspace. I also listen to music before I compete with my horses to get in the zone which may be why it worked well in this scenario.

Enjoy it!

Once I got into that interview groove I started to really enjoy them. I got to meet some incredible engineers and people while solving some fun and interesting problems. A lot of the code interviews I did felt like I was working with colleagues. Yes, I was responsible for coming up with the solution, but during the process I shared so many good laughs with interviewers as we commiserated over a typo or trying to put together a regex.

Maybe I got incredibly lucky and the 30+ people I met from the 6 companies I considered were all just really nice. Or, maybe some of it had to do with the mindset I went into these interviews with.

Once I had settled into interviewing, I approached each interview with a collaborative mindset. I was there to learn about the company as much as they wanted to learn about me. This made me much more relaxed and open during the interview and I think my interviewers ended up mirroring that feeling. If you go into an interview like you are heading into battle, then it likely will be a battle for both sides.

Even though I had to say no to many companies, they are still places I would have loved to work. I would recommend them to others in a heartbeat because of how much I enjoyed their interview process.

Good Luck!

Interviewing is not easy. It is a full on side hustle when you decide to take it on. But it is worth it in the end! Your perfect next opportunity is out there, now you just have to find it.

Anything else you all want to know about my interviewing process? Drop any additional questions you may have in the comments 👇

Incident Retro: Failing Comment Creation + Erroneous Push Notifications

Molly Struve (she/her) — Wed, 14 Jul 2021 20:53:41 +0000

This incident retro was tougher than most to share because, despite the seriousness of the issue, it affected only a very small percentage of our user base. However, we learned some incredibly valuable lessons and I think it's only right that I give others the chance to learn from our mistake as well.

Context

On Thursday July 8th, we merged a very large PR that updated our code to start using our new User Settings instead of the deprecated fields on users. The goal of moving these fields to user settings is to lighten up the user model and make these types of settings more configurable on a per-Forem basis.

Problem

On July 9th around midday, it was brought to our attention through a bug report that the comment creation experience on the frontend seemed to be broken. At this point folks on the team started looking into the problem. Using Honeycomb, we were able to confirm that the problem started when the User Settings PR was deployed which made it the prime suspect.

I immediately dove into the PR and quickly found the line of code that had been broken.

We had incorrectly removed the user_ids scope from the filtering used to collect user IDs for sending mobile push notifications. This was causing us to attempt to grab notification settings and IDs for all users with mobile notifications turned on which was often timing out. These timeouts would occur after a comment was created so the comment in most cases still existed but we never returned a success response to the frontend. This caused the frontend to freeze creating a bad experience for the user.

Fix

In order to remedy this situation, we quickly added back the scope and deployed. Immediately the errors and timeouts went away and things returned to normal for comment creation.

At this point, we concluded that possibly some extra notifications had been sent but that the majority of these requests had timed out so we assumed we were in a good state now. I looked for some sort of push notification model in our app where we might have stored unsent notifications but upon not finding one figured we were clear.

Fix 2

Upon realizing we had all of these records in Redis, we took the following steps to resolve the issue:

Removal of the iOS cert from Heroku to prevent sending any notifications. Without this cert we could ensure we would not send anymore bad notifications and that would give us time to fix the data in Redis.
To be safe, we removed all of the PushNotifications::DeliveryWorker jobs from Sidekiq so as not to be pinging Redis for the keys causing our key collection to slow down.
- At the same time, we looped through all of the keys in Redis and collected all of the rpush ones. These included all delivered and undelivered keys. There were 1.2 million rpush keys so this process took about 30 mins. We used the below script:

redis = Rails.cache.redis
start = 0
total = 0
key_collection = []
index, keys = redis.scan(start);
while index != "0"
  start = index
  total += keys.count
  keys.each do |key|
    key_collection << key if key.include?('rpush:notifications')
  end
  index, keys = redis.scan(start)
end

Once we had the keys collected we double checked that they were the keys we wanted and then we deleted them ALL in batches of 10 using this code:

key_collection.each_slice(10){|s| puts redis.del(s)}

Once the keys were gone, we double checked the count and then added back the iOS cert to Heroku.
After Heroku restarted, we ran a test to confirm that the notifications were back to sending and that we were recording delivered notifications correctly again in Redis.

Impact

Comment Creation

The comment creation flow on the frontend was broken for 30 hours after the UserSettings PR was deployed on July 8th. However, comment creation levels remained steady.

irb(main):001:0> Comment.where(created_at: 36.hours.ago..Time.now).count
=> 762
irb(main):002:0> Comment.where(created_at: 72.hours.ago..36.hours.ago).count
=> 857
irb(main):003:0> Comment.where(created_at: 108.hours.ago..72.hours.ago).count
=> 852
irb(main):004:0> Comment.where(created_at: 144.hours.ago..108.hours.ago).count
=> 756
irb(main):005:0> Comment.where(created_at: 180.hours.ago..144.hours.ago).count
=> 587

In total, there were 934 comments that were affected by the broken frontend interface.

Push Notifications

Given DEV is early on in its mobile journey, only 0.2% of our users have registered devices that are able to receive push notifications. This means that only 0.2% of our users were affected by this incident. We consider ourselves lucky that we were able to expose some of these issues and have this incident remain very contained.

Since the numbers of users affected was small, we decide to proactively reach out to all of those folks via email to apologize and explain why they might have received erroneous notifications.

Learnings

Large PRs

Large PRs naturally come with more risk. The bug that kicked off this stream of events was a small change in a very large PR that got missed by multiple folks. For starters, the diff for the line was not particularly helpful in recognizing the change that caused the issue. In addition, there were so many lines that it was easy to overlook. Breaking down PRs is one way to prevent this.

However, there are times when large, wide spread changes need to be made and in those cases you have to rely on your test suite.

Missing tests

The feature that broke was completely untested. Had we been testing that feature properly I think there is a good chance the bug could have been caught. Immediately after pushing the hotfix I added a test to ensure this never happens again.

Technical feature education

After fixing the bug, none of us working on the problem were well educated about how our Push Notification systems worked. Being late on a Friday, we skimmed the code and concluded we would be good to go. This ended up being the wrong assumption as we later learned Saturday that push notifications, unlike app ones, are stored and enqueued via Redis. I think it's more important than ever that we are somehow sharing and educating each other about some of these larger features as we roll them out.

Thankfully, we do have some great Push Notification documentation but it was never sought out during the incident. Is there a way we can make docs like this more visible? During the incident we were all heads down in the code, should we maybe have some sort of URL link in the code to the docs?

Being more intentional

I could have very easily reached out to our mobile team on Friday evening to double check that we were in a good state for Push Notifications. Once again, the whole Friday evening and wanting to be done caused me to accept my assumptions rather than challenge and check them.

One easy way to prevent something like this is to use a checklist. A checklist (heavily used in aviation because it has been shown to be key in preventing incidents,) is an easy way to make sure you never miss something and are deliberate and intentional with your decisions. We have great checklists for handling Heroku incidents in our internal Gitbook. However, we don't have a general incident on-call checklist which we plan to add.

Incident Response

Before I dive into specifics, I want to point out that this is the first larger incident we have had in a while. Incidents being rare is GREAT! But, it also means that our incident response was a bit rusty and our documented incident processes were a bit out of date to handle this situation. Going forward we will be taking a closer look at all of these things to ensure we are keeping them relevant and useful based on the current state of our application and team.

Broken Comment Creation Reporting Flow

The initial comment creation problem was reported by a DEV user via a GitHub issue and seen about 11 hours later by our internal engineering team. GitHub Issues are naturally not a very urgent form of reporting for us since they are handled during working hours. One way we could improve this flow would be to further communicate that urgent support issues need to be emailed to yo@forem.com. Our customer success team is always monitoring these channels diligently and could more quickly triage and escalate an issue to the team.

Manual reporting aside, ideally, we should have caught this programmatically with our monitoring by alerting on the increased HTTP request errors that were detected.

Going forward, we plan to set up a monitor to alert us for these changes so that we are the first to find out, rather than relying on users reporting the issue to us. With the proper monitor in place, we could have caught this within a couple of hours instead of over a day.

Broken Push Notifications Reporting Flow

We responded to the broken comment creation issue immediately when it was surfaced, however, the followup push notification issues were not. Our engineer correctly pinged the #emergency channel when he realized that we were having problems with notifications but did not escalate the issue and wake up the on-call dev via PagerDuty.

One way we could streamline this process would be to see if we could set up our #emergency Slack channel to automatically ping PageDuty when a message is posted. Another option is address some of our incident response rustiness by better educating the team on our desired incident response flows. This probably needs to be a reoccurring training that happens periodically so as to prevent processes and knowledge from getting stale as our systems evolve.

Technical Remediation

One delay we encountered when trying to fix this issue was that the Rpush keys were stored in the same Redis instance as our Rails cache. Given our Rails cache is massive and contains 4+ million keys, looping through all of them to find only the Rpush keys was tedious and took a while. That cache is also very active which caused us to hit some timeouts when we were trying to perform heavier operations on it.

We could have mitigated this friction by having either a separate Redis instance for Rpush OR by having those keys in a separate database away from our Rails cache keys. Going forward, as we grow our mobile platforms, I think we should make one of the above changes to ensure that we have easy access to these keys and records. It will also ensure that as we grow and scale push notifications we don't have to worry about impacting other systems like our core Rails app.

Thanks Everyone 🤗

A lot of people gave up their personal time to come together on Friday and Saturday to mitigate and fix these issues. I truly appreciate the amazing team we have at Forem and their dedication to this software. We learned a lot about our systems in the process which ensures that this incident will not go to waste. I hope others can learn from this as well.

Goodbye Offline Page

Molly Struve (she/her) — Wed, 17 Mar 2021 17:53:35 +0000

Dear DEV Community Members,

I wanted to bring you all up to speed on a decision the Forem team recently made to remove our Service Worker functionality, and thus, our offline page. This was not a decision we came to lightly.

Like many of you, we all thoroughly enjoyed drawing in our browsers using the DEV offline page whenever our internet fritzed on us. Recently however, the cost of maintaining the Service Worker became exceedingly high for our team. Over the past couple of months, we've been having to put out one emergency fire after another, all because of our Service Worker functionality.

A couple of weeks ago, I took a step back and assessed the situation. Given the fact that we lean heavily into edge caching, I figured that losing the Service Worker would not have a huge impact on performance and could save the Forem engineering team a lot of headaches. However, there still remained the question of the offline page. After chatting with @ben about our options, we decided to strip the Service Worker down so that its only function was to serve the offline page.

Reduce service worker functionality to minimal offline page #12834

benhalpern posted on Feb 26, 2021

What type of PR is this? (check all applicable)

[x] Refactor
[ ] Feature
[x] Bug Fix
[ ] Optimization
[ ] Documentation Update

Description

This PR strips out some of our very cool ReadableStream service worker functionality because it has been causing too many bugs related to the deployment of new code.

The upside of this functionality is very instant initial page loads, but the downside to the current implementation is too many instances of difficult-to-predict deployment cache issues. These were always painful to deal with, but are exceptionally difficult in a Forem world where our devops cannot act quickly to fix production issues the same way we can with a single deploy.

The fundamental issue with our current implementation is just that it is not exhaustive in terms of dealing with certain scenarios, and dealing with edge cases creates a lot of complexity.

We suspect there are still UX benefits and SEO benefits to handling basic offline functionality...

And we still have our service worker lifecycle on top of which we may be able to methodically add back certain functionality that this PR removes, but we can do so with much more of an emphasis on ensuring all possible scenarios are accounted for so that we do not allow users to get caught in buggy situations.

One additional adjustment I made was just removing the "DEV" portion of the image on the offline page (and some code cleanup there)... We may want to offer a new fun way for Forem admins to customize this page, but I figured this small adjustment to make for a more minimal and generalized offline page made sense here.

Follow up

There is additional code that can be removed with this new approach, but due to the nature of how service workers run code already downloaded to user browsers (i.e. the cause of this complexity in the first place), it makes sense to wait at least a few weeks to remove some of the other code that is not needed anymore in case it could break old installs.

Related Tickets & Documents

https://github.com/forem/internalEngineering/issues/333

QA Instructions, Screenshots, Recordings

This is mostly an adoption of this functionality: https://web.dev/offline-fallback-page/

Eyeball the functionality and test different network conditions to ensure this does not cause new problems.

UI accessibility concerns?

This should not cause new a11y issues.

Added tests?

[ ] Yes
[x] No, and this is why: I'm not really sure how to test this.
[ ] I need help with writing tests

[Forem core team only] How will this change be communicated?

Will this PR introduce a change that impacts Forem members or creators, the development process, or any of our internal teams? If so, please note how you will share this change with the people who need to know about it.

[x] I've updated the [Developer Docs]

[optional] What gif best describes this PR or how it makes you feel?

View on GitHub

We deployed and rejoiced thinking our woes were over.

We were wrong

A few days after this deploy, our emergency Slack channel was once again lit up with users who could not log in from Safari. We put together a fix (and then a follow up fix 🙈) and shipped it. A few days after that, the emergency channel again popped up with a flood of log in problems being reported from users. Once again we shipped a fix. After the third day of this happening, I decided to turn to my engineering leaders to get their thoughts on the whole situation.

What are folks' thoughts on the continuing issues we are having with the Service Worker? Ben greatly reduced its functionality to only the offline page bc of all the issues it has been causing and the giant sink it has been for developer time but it doesn't feel like things have gotten much better. My heart can't take many more days of coming back to my Slack with #emergency lit up and a 100 reply thread.

Do we think things are going to get better once we have identified all the exception paths? OR do we still think it's going to be a landmine that will only continue to bite us? The cost of this offline page feature is getting pretty high and I am wondering if it's worth the cost.

My incredible team and @ben all jumped in with their thoughts.

But there was another complication thrown into the mix: our offline page was supposed to help our SEO rankings! However, we were not sure how much it was supposed to help. In the end, we weighed all our options and determined that the best course of action was to remove the Service Worker completely and monitor our incoming traffic using Google's Search Console.

Making engineering decisions is all about tradeoffs. In this case, the Service Worker feature was disruptive to both our users and to us as developers, which seemed to outweigh any benefits that it had provided. While removing the offline page is a bit disappointing, in the big scheme of things, it will make the Forem platform much more reliable and stable going forward.

For more details, check out the removal PR below. 👇

Removed service worker #12974

nickytonline posted on Mar 11, 2021

What type of PR is this? (check all applicable)

[ ] Refactor
[ ] Feature
[x] Bug Fix
[ ] Optimization
[ ] Documentation Update

Description

This removes all of our service worker functionality with a self-destructing service worker which appears to be the best approach for unregistering a service worker. See https://github.com/NekR/self-destroying-sw#how-to-use.

Related Tickets & Documents

QA Instructions, Screenshots, Recordings

Test this in Chrome, Firefox, and Safari. Note in Safari, there may be no service worker registered due to the work done with the service-companion.js file. In that case, ensure that no errors are thrown in the console of the web dev tools in Safari.

Pull down latest of the main branch and deploy the site locally. Navigate to the home page and validate that the service worker is running by opening the developer tools and navigating to the developer tools.

Chrome

Firefox

Safari

Pull down the branch for this PR and deploy it locally. Refresh the browser window that is currently open and validate that the service worker gets unregistered.

Chrome

Firefox

Safari

UI accessibility concerns?

N/A this is front-end infrastructure

Added tests?

[ ] Yes
[x] No, and this is why: I've removed tests related to the service worker, so existing tests should continue to pass.
[ ] I need help with writing tests

[Forem core team only] How will this change be communicated?

[ ] I've updated the Developer Docs and/or Admin Guide, or Storybook (for Crayons components)
[ ] I've updated the README or added inline documentation
[x] I will share this change in a Changelog or in a forem.dev post
[ ] I will share this change internally with the appropriate teams
[ ] I'm not sure how best to communicate this change and need help
[ ] This change does not need to be communicated, and this is why not: please replace this line with details on why this change doesn't need to be shared

[optional] Are there any post deployment tasks we need to perform?

[optional] What gif best describes this PR or how it makes you feel?

</div>
<div class="gh-btn-container"><a class="gh-btn" href="https://github.com/forem/forem/pull/12974">View on GitHub</a></div>

Duplicate Digest Email Incident Retro From January

Molly Struve (she/her) — Tue, 16 Mar 2021 18:13:20 +0000

Sorry all that this incident retro is a couple of months late. We did the retro internally immediately after the incident occurred and then I put it on my TODO list to make it public on DEV and that's where it stayed for far too long. 😬

Background Context

Over the past few months, we have been working hard to make the Forem setup more configurable for our creators. In an effort to do this we recently updated how we create a Forem's community name. This was done by merging the following PR that appended the collective_noun, if it was not disabled for the particular Forem, to the SiteConfig.community_name through a DataUpdateScript:

Adds collective_noun Fields Back to SiteConfig Model #12055

juliannatetreault posted on Dec 28, 2020

What type of PR is this? (check all applicable)

[ ] Refactor
[ ] Feature
[x] Bug Fix
[ ] Optimization
[ ] Documentation Update

Description

This PR adds the collective_noun and collective_noun_disabled columns back to the SiteConfig model after running into a data_update script failure in another PR. This is necessary so that when running the data_update script responsible for updating the community_name, we have access to the collective_noun and collective_noun_disabled methods. Additionally, this PR adds a data_update script that conditionally updates SiteConfig.community_name if SiteConfig.collective_noun_disabled returns false. If, however, the forem has collective_noun_disabled set to true, then the community_name will remain as-is and untouched.

Related Tickets & Documents

Relates to PRs #11846 , PR #11973 , and PR #12054

QA Instructions, Screenshots, Recordings

To QA these changes, first ensure that all checks pass and the Travis build is green, then:

Run the data_update script manually and ensure that your Community's name updates accordingly

UI accessibility concerns?

N/A

Added tests?

[x] Yes
[ ] No, and this is why: please replace this line with details on why tests have not been included
[ ] I need help with writing tests

Added to documentation?

[ ] Developer Docs and/or Admin Guide
[ ] README
[x] No documentation needed

[optional] Are there any post-deployment tasks we need to perform?

The data_update script included in this PR needs to successfully run!

[optional] What gif best describes this PR or how it makes you feel?

View on GitHub

Data Update Script:

module DataUpdateScripts
  class AppendCollectiveNounToCommunityName
    def run
      return if SiteConfig.collective_noun_disabled || SiteConfig.collective_noun.blank?

      SiteConfig.community_name = "#{SiteConfig.community_name} #{SiteConfig.collective_noun}"
    end
  end
end

The PR nicely addresses and updates all places in the code where we were using community_name and therefore went out without any problems.

However, 4 days later on Wednesday, December, 30th, we began to get reports of folks receiving duplicate digest emails.

// Detect dark theme var iframe = document.getElementById('tweet-1344320876741861377-288'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1344320876741861377&theme=dark" }

I quickly jumped in and determined the problem to be an Emails::EnqueueDigestWorker that was continuously being restarted by deploys and sending the duplicate digest emails.

Bug Trigger*

NOTE: I did not label this section "Cause" because I believe the real, root cause of this problem came much earlier in the year. Scroll down for more details.

The Emails::EnqueueDigestWorker is a worker that we have "turned off" for DEV because it takes too long to complete given the high number of users we have. We know it will keep restarting on deploys so we choose to use a rake task to send digest emails. To keep the worker from running we have a guard clause at the beginning of the worker that tells it to exit early if the environment is DEV. The guard clause at the time of the incident looked like this:

return if SiteConfig.community_name == "DEV"

The problem was, when the above PR was deployed we changed community_name from "DEV" to "DEV Community" causing this guard clause to no longer work which allowed the worker to start running.

Mitigation

The first thing I did to mitigate the number of those affected by this problem was to clear all of the existing digest worker jobs out of Sidekiq. Once that was done, I pushed out a quick fix to update the hardcoded string to the current community name by adding "Community" to it.

Bug Fix:Update dev.to check to use DEV Community #12082

mstruve posted on Dec 30, 2020

Recently we changed our site name on dev.to from DEV to DEV Community. This is used in our code in a couple of places and needs to be updated. From a prod console

irb(main):002:0> SiteConfig.community_name == "DEV Community"
=> true

Without the guard clauses working properly we enqueued our Email Digest worker which takes hours to complete for DEV bc of all the users. Since we restart Sidekiq on every deployment, this worker was continuously restarting which is why the duplicate emails were getting sent.

View on GitHub

While this fixed the immediate issue, the hardcoded string is what got us into this jam in the first place. To prevent future problems I quickly followed up with a second, more resilient PR, that used a more robust method to check for the DEV environment.

Refactor:Use dev_to? to Check for DEV #12083

mstruve posted on Dec 30, 2020

What type of PR is this? (check all applicable)

[x] Refactor

Description

Rather than checking the community name which we just found out can change easily, lets check the domain like we do in other places since that is far less likely to change.

Related Tickets & Documents

https://github.com/forem/forem/pull/12082

View on GitHub

Root Cause and Learnings

Even though the the community name PR was the trigger for this issue, it was far from the root cause. This issue was really caused by the poor decision by myself to hard code this string in the first place.

Bug Fix:Run Individual Digest Workers inline for DEV #10070

mstruve posted on Aug 29, 2020

What type of PR is this? (check all applicable)

[x] Bug Fix

Description

Followup to https://github.com/forem/forem/pull/10065, we need to also ensure the individual workers run inline and are not piling up in Sidekiq since running them in parallel is what is stressing the database.

Added tests?

[x] yes

View on GitHub

Hardcoded strings, especially those for variables like this that can change, should be avoided at all costs when writing code. At the very least you should be writing a method or using a CONSTANT to represent the string you want to check. This allows you to easily Find/Replace that method or CONSTANT in the future so it doesn't get lost in the shuffle as it did here.

The second problem was that I deployed that fix without a PR review. I announced the problem in Slack and linked the PR, but based on the Slack message our database was in a dire state which is why I chose to expedite the fix. In the future, trying to get at least 1 PR review should always be the goal even if it means waiting an extra minute or two. Even if you don't immediately take the advice of the PR review, at least you can go back and update things later.

The first fix that I pushed out in order to quickly update the digest worker worked but was un-ideal. This was pointed on in the PR review. I was still able to quickly push the hotfix but then immediately went back and wrote more thoughtful and robust code after. That flow is how the initial bug should have been handled. Our recently renewed commitment to sticking more to our processes now is what led to us fixing this problem correctly the second time around, while the first time we left the door open for future problems.

Finally, the more we can move away from this "DEV is a special snowflake" concept the better. If every environment is treated the same then you don't have to worry about edge cases popping up unexpectedly in production for specific environments. The final goal is to remove that guard clause altogether so every Forem sends its email digests the same way. Once again, this eliminates another special case solution that lowers the risk of future surprises.

Handling Hotfixes

Always try to get a PR review even for hotfixes! Most of us are not running life-saving software. If your site remains down for an extra min or two because you did the responsible thing and got a code review, it's not the end of the world. Having a good code review will decrease your chances of shipping unreliable code that could cause future issues.

If a fix MUST go out immediately and you cannot get a review beforehand(maybe it's super late and no one is online), then get a follow-up review. Ask someone to review the code after the fact to make sure it is sound and is not so hacky as to lead to future problems.

TL;DR

Don't hard code strings and get a PR review! And if all else fails, learn from your mistakes and strive not to make them again. Happy coding!

Welcoming Katie Davis and Suzanne Aitchison to the Forem Team!

Molly Struve (she/her) — Thu, 14 Jan 2021 22:47:27 +0000

We are very excited to start off 2021 with the addition of two new staff members who will focus on Forem's front end development. Please meet Katie Davis (Senior Software Engineer) and Suzanne Aitchison (Software Engineer).

Katie and Suzanne will work closely with Senior Software Engineer, @nickytonline and Head of Engineering, @molly to support the ongoing development and maintenance of dev.to, Forem, as well as other apps and projects.

Our Forem rollout is off to a fast-moving start, and we are thrilled to have such highly-skilled additions to our tight-knit team.

Please give a special welcome to Katie and Suzanne and check out their introductory posts.

Hi! I'm Joining the Forem Team 🎉

Katie Davis ・ Jan 14 '21

#meta #career #forem

I joined the Forem team!

Suzanne Aitchison ・ Jan 14 '21

#meta #career #forem

What's the best or worst recruitment line you have heard?

Molly Struve (she/her) — Fri, 18 Sep 2020 22:37:17 +0000

After reading yet another pesky, eye-rolling recruiter email, I started wondering, what kinds of emails others get. Share the best or worst lines you have ever received from a recruiter!

Rake::Task .enhance() Method Explained

Molly Struve (she/her) — Wed, 05 Aug 2020 18:33:09 +0000

I recently fixed a bug in our Forem software that led to some serious Ruby learnings that I have to share.

The Bug

The problem we were having was that a Postgres view we use to expose data to our team, called hypershield, was not getting refreshed correctly. Ideally, we want this view refreshed AFTER we run migrations so that it is up to date with the database. However, it came to my attention when looking at our logs that the hypershield view was being refreshed BEFORE we ran migrations.

[hypershield] Refreshing schemas
[hypershield] Success!
== 20200726215928 ChangeTagIdsToBigints: migrating ============================
== 20200726215928 ChangeTagIdsToBigints: migrated (4.9509s) ===================

This led to our hypershield views being out of date with our actual database.

.enhance()

To ensure that the hypershield view is refreshed when we migrate our database we use the Rake::Task method .enhance().

Rake::Task["db:prepare"].enhance(["hypershield:refresh"])

In the past when I used the method enhance it always ran the additional rake task AFTER the task I was "enhancing". This got me really confused as to why the behavior was suddenly different, so I went digging.

During my digging, I came across the Rake::Task docs. Here, I opened up the source code for the method.

def enhance(deps=nil, &block)
  @prerequisites |= deps if deps
  @actions << block if block_given?
  self
end

The first thing that struck me was that the behavior was different if you passed in an argument versus a block.

When passed an argument, that argument became one of the @prerequisites for the task, meaning it was run BEFORE. When passed a block, the block was added to a list of @actions. According to the docs, @actions are "attached to a task", meaning they run AFTER the task.

The Fix

If I want to refresh our hypershield view after we run migrations I need to pass that refresh task to enhance in a block and not as an argument. The final fix looks like this:

Rake::Task["db:prepare"].enhance do
  Rake::Task["hypershield:refresh"].execute
end

Now, the hypershield view is updated AFTER migrations are run. This ensures the view is always up to date with our database. NOTE the .execute that we have to use to invoke our task. This is not needed when you pass the task as an argument, but it is needed when you are using a block.

TL;DR

Passing a task as an argument to enhance causes it to run BEFORE the task you are "enhancing".

Rake::Task["task_A"].enhance(["task_B"])
# Runs task_B
# Runs task_A

Passing a task to enhance in a block causes it to run AFTER the task you are "enhancing".

Rake::Task["task_A"].enhance do
  Rake::Task["task_B"].execute
end
# Runs task_A
# Runs task_B
```



Now go and enhance away!!!

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/6tuyrapc49w9prajk3va.gif)

Share Your Best Typo Story!

Molly Struve (she/her) — Thu, 25 Jun 2020 23:11:56 +0000

I recently posted a tweet about fixing a bug that turned out to be a misspelled word. This has happened to me more times than I care to admit 😅

// Detect dark theme var iframe = document.getElementById('tweet-1276195170292322311-566'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1276195170292322311&theme=dark" }

Anyways, all of the replies I can totally relate to, and now I want to hear more! What is your best typo story? Is there a word you can never get right? Did you spend hours debugging something that turned out to be a typo? Let's hear what ya got because we have all been there!

Using KnapsackPro to Parallelize Your Tests

Molly Struve (she/her) — Fri, 19 Jun 2020 14:20:37 +0000

Recently, DEV was struggling with CI build times. It got to the point where Travis builds were taking up to 30 mins to complete. On top of that, we had a few flaky specs in our build and the result was some very frustrated developers.

When I began trying to figure out how to solve this issue, the first strategy that came to mind was parallelizing the build process. This means splitting up our test suite into chunks and running each of those chunks at the same time. The challenge with this is figuring out how to split the chunks up so that each chunk runs in the same amount of time.

For example, if you have a 30 min build and you want to split it into 3 parallel builds, ideally you want each build to run for 10 min. In order to do this, you have to figure out how long each of your tests takes to run. While grappling with this problem, one of my coworkers suggested I checkout KnapsackPro.

KnapsackPro

NOTE: KnapsackPro has two modes, Queue Mode and Regular Mode. Currently, DEV is using Regular Mode and that is what I will be covering in the rest of this blog post.

KnapsackPro allows you to evenly split up your tests between parallel CI builds so that they run in the most optimal way to save you time. KnapsackPro does this by recording the time each test takes to run. It then uses that timing data to split your tests up equally in terms of runtime into however many groups you choose. This sounded like a great solution so I started digging into the docs trying to figure out how to get it all setup

Getting Started: KnapsackPro and Ruby

Before I go any further, I have to say that the KnapsackPro docs are some of the best I have worked with. They offer a thorough step by step setup plan for whatever language or testing framework you are using. In addition, their FAQ docs cover just about every possible buggy scenario you might run into. All of this made setting up KnapsackPro a straight forward process.

Here are the steps to setup KnapsackPro with a Rails project. I pulled these straight out of the KnapsackPro gem installation guide.

Step 1: Install the Gem

Add these lines to your application's Gemfile and then run bundle install.

group :test, :development do
  gem 'knapsack_pro'
end

Step 2: Customize Setup Based on Tests

Once you have the gem installed then its time to set up your configuration based on what kind of testing framework you use. To figure out how to configure the gem, KnapsackPro gives you this handy installation guide.

Simply select your testing frameworks and gems and it will tell you what to add to your configuration. In our case, we use Rspec, Webmock, and TravisCI which gave us these additional steps to perform.

RSpec Setup

Add the following at the beginning of your spec/rails_helper.rb or spec/spec_helper.rb:

require 'knapsack_pro'
KnapsackPro::Adapters::RSpecAdapter.bind

VCR and Webmock Configuration

We use VCR and Webmock so we needed to add the Knapsack Pro API subdomain to ignore hosts for those configurations.

require 'vcr'
VCR.configure do |config|
  config.hook_into :webmock # or :fakeweb
  config.ignore_hosts('localhost', '127.0.0.1', '0.0.0.0', 'api.knapsackpro.com')
end

# add below when you hook into webmock
require 'webmock/rspec'
WebMock.disable_net_connect!(allow_localhost: true, allow: ['api.knapsackpro.com'])

To make sure everything loads properly, ensure you have require false for your webmock gem when VCR is hooked into it.

group :test do
  gem 'vcr'
  gem 'webmock', require: false
end

The docs also state:

If you happen to see your tests failing due to WebMock not allowing requests to Knapsack Pro API it means you probably reconfigure WebMock in some of your tests. For instance, you may use WebMock.reset! or it's called automatically in the after(:each) block, if you require 'webmock/rspec'. These setups will remove api.knapsackpro.com from allowed domains. Please try below to fix this issue:

RSpec.configure do |config|
  config.after(:suite) do
    WebMock.disable_net_connect!(
      allow_localhost: true,
      allow: [
        'api.knapsackpro.com',
      ],
    )
  end
end

Travis CI Setup

Using the Travis matrix feature we were able to parallelize our builds on Travis with the following updates to our .travis.yml file.

script:
  - "bundle exec rake knapsack_pro:rspec"
env:
  global:
    # tokens should be set in travis settings in web interface to avoid expose tokens in build logs
    - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=rspec-token
    - KNAPSACK_PRO_CI_NODE_TOTAL=3
  jobs:
    - KNAPSACK_PRO_CI_NODE_INDEX=0
    - KNAPSACK_PRO_CI_NODE_INDEX=1
    - KNAPSACK_PRO_CI_NODE_INDEX=2

Below is the DEV Pull Request that made all of these changes in our repository.

Spec Speedup: Use Knapsack to Run Parallel Builds #8390

mstruve posted on Jun 10, 2020

What type of PR is this? (check all applicable)

[x] Optimization

Description

This PR introduces a service called Knapsack to help us parallelize our spec suite as evenly as possible. The first time I ran Knapsack in our build it ran every test separately and recorded the time it took to run the test. Using this information Knapsack then splits the tests up for us into 3 equally timed groups to run in parallel each time we run our test suite. This is how regular mode works.

The changes in this PR are introducing the gem and setting it up. All of them were made with the help of the Knapsack installation guide which walks you through all the changes you should make to get it working properly.

NOTE - There is currently a bug with the parallelization in Travis that causes the --local flag for our bundler command to be ignored. This means on your first build, since there is no travis cache, the jobs will likely take 13min. I am in contact with Travis support to get this resolved.

Why aren't we using Queue Mode? Ideally, we want to use queue mode. In Queue Mode Knapsack sends us groups of 3-5 specs at once and then when they finish, sends another group of specs. It keeps doing this until all specs have been run. This is obviously the fastest approach but we ran into some errors with the jobs hanging. My goal is to get the regular version out then try to debug that hanging issue so we can use queue mode.

How much does Knapsack cost? FREE bc we are open source and I must say the founder has been extremely helpful in getting us going and holding my hand through the integration process.

@snackattas

View on GitHub

If you click through to the pull request, you will notice we also made some additional changes to ensure our code coverage checks and other CI steps ran efficiently and correctly with our new parallel builds.

Using the Gem

With all of those changes in place, you will then want to push your branch up and let the KnapsackPro API do its thing. Keep in mind, the first run will NOT be optimal because the knapsack_pro gem will record the execution time of every one of your tests.

To make sure everything was recorded successfully, you can check the build metrics on your KnapsackPro API dashboard.

Here you can find everything from node build times to a breakdown of how long each test took to run. Once KnapsackPro has that data, then it can strategically split your tests up as evenly as possible for all future builds. Your second test suite run on your CI provider will be parallelized with the optimal test suite split if the first run was recorded correctly.

Gotchas

Setting Up KnapsackPro for Forks

One hiccup we ran into when implementing KnapsackPro was that it would not work for forks because forks do not have access to our KnapsackPro tokens in Travis. We ended up seeing this error

Missing environment variable KNAPSACK_PRO_TEST_SUITE_TOKEN. You should set environment variable like KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC (note there is suffix _RSPEC at the end). knapsack_pro gem will set KNAPSACK_PRO_TEST_SUITE_TOKEN based on KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC value. If you use other test runner than RSpec then use proper suffix.

Once again, the great KnapsackPro docs came to the rescue with a section in the FAQ that explained how to get KnapsackPro to work with forked branches.

The TL;DR of the solution is that we had to create an executable file bin/knapsack_pro_rspec in our main project repository to handle the missing tokens.

#!/bin/bash
if [ "$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC" = "" ]; then
  KNAPSACK_PRO_ENDPOINT=https://api-disabled-for-fork.knapsackpro.com \
    KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=disabled-for-fork \
    bundle exec rake knapsack_pro:rspec 
else
    # Regular Mode
    bundle exec rake knapsack_pro:rspec
fi

Then, in our .tavis.yml file, we replaced bundle exec rake knapsack_pro:rspec with bin/knapsack_pro_rspec. You can see the changes in this PR:

Allow Knapsack to work for Forks #8413

mstruve posted on Jun 11, 2020

In order to get KnapsackPro to work on forks we need to use this workaround from the Knapsack docs

View on GitHub

The new script tries to hit the KnapsackPro API, but without the token, it fails. Upon failure, it will fallback on grouping tests by directory names and you will see an output that looks like this:

W, [2020-06-17T08:45:14.412458 #8343]  WARN -- : [knapsack_pro] Next request in 2s...
W, [2020-06-17T08:45:16.577365 #8343]  WARN -- : [knapsack_pro] #<SocketError: Failed to open TCP connection to api-disabled-for-fork.knapsackpro.com:443 (getaddrinfo: Name or service not known)>
W, [2020-06-17T08:45:16.577552 #8343]  WARN -- : [knapsack_pro] Fallback mode started. We could not connect with Knapsack Pro API. Your tests will be executed based on directory names. Read more about fallback mode at https://github.com/KnapsackPro/knapsack_pro-ruby#what-happens-when-knapsack-pro-api-is-not-availablenot-reachable-temporarily

Grouping by directory name is not quite as ideal as grouping by timing, but it beats not parallelizing things at all.

Why KnapsackPro is AWESOME!

Great Documentation

I said it above and I will say it again, KnapsackPro is extremely well documented which makes getting started with it very straight forward. There is literally a doc for just about every question or scenario you can run into.

Public Dashboard

One of the big benefits of KnapsackPro is that they give you the option to make your dashboard and test stats public. This is a huge deal for us at DEV because we have so many external contributors. It is amazing when those contributors can access the same data that the core team can.

Customer Service is Beyond Good

KnapsackPro is a small company, which means when you send them a support email it goes straight to a real person! No automated response, no bouncing around between different support people with canned responses. You go straight to someone who will be able to help you.

At the start of this, DEV's test suite was in pretty rough shape in terms of flakiness and reliability. It's also worth mentioning that I am a Site Reliability Engineer, not a QA engineer, so I struggled quite a bit getting everything setup. What got me through was the support and help I received from KnapsackPro along the way. Email responses were quick(within 24 hrs) and not only would they answer my questions, but they also offered me tips about how to set things up even more efficiently than I was.

End Result

The end result of all this work is that we now have a test suite that runs in about 10 min! In addition, when we come across a new flaky spec, we can simply retry the one job that failed, instead of having to run the entire suite.

The move also forced us to separate our testing process and our deploy process. Now when a deploy fails for some external reason, we can simply retry the deploy step. Before, we would have to restart the entire build and run the whole test suite again before we could deploy. It was not fun.

Devs are always looking for a better and faster CI and KnapsackPro is a great tool that can help you accomplish that.

I was not enticed or asked to write this blog post by anyone from KnapsackPro. I know many companies struggle with slow test suites and I wanted to share how we tackled that problem at DEV so hopefully, others might be able to do what we did to solve their own challenges.

Tips for Running Scripts in Production

Molly Struve (she/her) — Tue, 16 Jun 2020 16:32:49 +0000

I know, how dare I suggest running a script in production. I am a Site Reliability Engineer, I should never condone such craziness. But truth is, there will likely come a time when you need to run a script in production to update or cleanup some data. In this post I am going to give you some tips about how to write and execute a script in production as safely as possible.

1) Track Your Progress

Nothing is worse than writing a giant block of code, pasting it into a console, then hitting enter and watching it sit there. You have no idea where the code is in the script or what it is doing and that, at least for me, is terrifying.

For this reason, you always want to make sure you output some sort of progress meter from your scripts. This allows you to follow along and know where you are in your process. In the event you are using Ruby, consider some well placed puts statements. Below is a script that we recently used at DEV to clean up some incorrectly cached data. Notice the puts statements throughout the script that allow us to follow along as it does its work.

invalid_articles = []
Tag.where(id: tag_ids).find_each do |tag|
  puts tag.taggings.count

  tag.taggings.each_with_index do |tagging, index|
    puts index if index%100 == 0
    article = tagging.taggable
    next unless article

    result = article.update(cached_tag_list: article.tags.pluck(:name).join(", "))
    if result
      puts "Artcle update success #{article.id}"
    else
      puts "Artcle update failure #{article.id}"
      invalid_articles << article
    end
  end
end

Also notice that we are keeping track of any invalid articles that we might find while running this script. Especially when you are cleaning up bad data, always assume you might stumble across more of it and prepare for that in your script. Here we use an if/else statement to catch any invalid articles. You could also use a begin/rescue block.

2) Record Before and After States

When you are updating records there is always a chance something will go off the rails. In order to have the ability to "roll back" track your before state as you are making the updates. If we update our script above to do this, here is what it would look like.

invalid_articles = []
before_update_tag_lists = {} 
Tag.where(id: tag_ids).find_each do |tag|
  tag.taggings.each_with_index do |tagging, index|
    puts index if index%100 == 0
    article = tagging.taggable
    next unless article
    # Record the current cached tag list for every article
    before_update_tag_lists[article.id] = article.cached_tag_list

    result = article.update(cached_tag_list: article.tags.pluck(:name).join(", "))
    if result
      puts "Artcle update success #{article.id}"
    else
      puts "Artcle update failure #{article.id}"
      invalid_articles << article
    end
  end
end

If anything goes wrong while this script is running, the before_update_tag_lists hash has all of our original data in it. Using this original data we can loop back through the articles and reupdate them with the old lists if necessary.

3) Write Production Quality Code

It can be tempting when you are writing a script to use as little syntax as possible. Usually, this means throwing in single letter variables everywhere. You probably won't ever use this code again, so why waste time making it look pretty and readable? The reason you want to make it pretty and readable is because then the script is easier to understand and follow. Having a script that is easy to understand will help you avoid writing bugs.

In my script example above I clearly write out what each object is that I am working with. This allows nearly anyone to look at the script and be able to understand what it is doing. This leads me to my next script writing tip.

4) Have Your Script Reviewed

The same way you never want to push code out to production without a code review, you shouldn't run a script in production without a code review. This is another reason why you want to make sure your script is understandable and readable, because you want someone else to be able to also figure out what it is doing.

We all know the value a second set of eyes on our code brings. Even if you find yourself in a situation where time is tight and you need to run a script ASAP, try as hard as you can to get a second set of eyes on it. I can't tell you the number of times a fresh set of eyes has kept me from botching a script update.

5) Use Screen or Tmux for Long Running Scripts

Tmux and Screen allow you to start an ssh session in a shell and keep that shell active even through network disruptions. This ensures that if you lose connection while your script is running, the script run will not be interrupted. Thanks @kinduff for the reminder!

Alejandro AR • Jun 17 '20

If the task takes a good amount of time, you have risks of being disconnected from either the SSH session, your internet provider, etc.

To avoid this, I recommended running these scripts (although, I do not recommend running scripts like this at all) using screen. It's super easy to use:

SSH into the desired instance
Start a new screen session using screen
Run the long running script
Press Ctrl + a followed by d to detach the session
You can now close everything, even the SSH session
You can reattach to the screen session using screen -r

Screen has a lot of awesome things, make sure to check out the man page.

Running a script in production is never ideal, but if you use these tips when you do it, it can make the experience much less daunting.

Happy scripting!

Optimizing Code as You Go

Molly Struve (she/her) — Tue, 02 Jun 2020 16:25:55 +0000

I am a big proponent of always thinking about scale and optimization when you are writing code, no matter how big or small of a product you are working on. A lot of people like to argue that there is no reason to optimize when you don't have to and that is wastes time. Yes, sometimes it does take a little bit of extra time, but down the road that little bit of time can save you a huge headache.

I'm not talking about rearchitecting a whole feature, I'm talking about the little things you can do while you are coding that I guarantee will make a difference in the future. In this post, I will walk through an example of a small optimization I made recently at DEV with this pull request and how we could of avoided creating that pull request altogether.

The Most Popular DEV endpoint

Every time a DEV page is loaded we make an asynchronous call to get some user data to help us populate variables in the view. This user data comes from the endpoint async_info/base_data. Because this call is made for every page load it is above and beyond our most hit endpoint. The top purple line shows the number of calls made to our async_info/base_data endpoint.

Given we call this endpoint a lot I decided to take a closer look at it to ensure it was as lean and mean as it could be. The first thing I did was checkout how we were building the hash we were returning. Here is what the hash looked like before my change:

{
  id: @user.id,
  name: @user.name,
  username: @user.username,
  profile_image_90: ProfileImage.new(@user).get(width: 90),

  followed_tag_names: @user.cached_followed_tag_names,
  followed_tags: @user.cached_followed_tags.to_json(only: %i[id name bg_color_hex text_color_hex hotness_score], methods: [:points]),

  followed_user_ids: @user.cached_following_users_ids,
  followed_organization_ids: @user.cached_following_organizations_ids,
  ...
}

Notice we have two different tag fields returning the same data. I do not know which one field was added first, but for the sake of this example lets say that followed_tag_names was their first.

Let's pretend...

Let's pretend you are the past developer adding the followed_tags value to the hash. You need all of the tag information in the view and not just the names, so you add the new field followed_tags to the hash. Here, is where many developers stop. They have the information they need so their work is done.

This is when I say, don't stop! Take a moment and look at your hash. You are now making TWO calls to get the same Tag information. While this may be fine when the hash is not created often, it is far from ideal when your site starts to take off and this hash is created millions of times a day. That extra call will add up. If instead of stopping, you take a few extra minutes to address this unoptimal code, you can easily solve this issue and save your future self a lot of headache.

Optimization Options

Here are the two options I considered when I was optimizing this code.

1) We could memoize the tags in a variable and then reference that variable in our hash like below. This will save us that extra database hit and ensure we don't waste resources on redundant calls.

tags = @user.cached_followed_tags
{
  id: @user.id,
  name: @user.name,
  username: @user.username,
  profile_image_90: ProfileImage.new(@user).get(width: 90),
  followed_tag_names: tags.map(&:name),
  followed_tags: tags.to_json(only: %i[id name bg_color_hex text_color_hex hotness_score], methods: [:points]),
  followed_user_ids: @user.cached_following_users_ids,
  followed_organization_ids: @user.cached_following_organizations_ids,
  ...
}

2) If we don't want to make the extra database call AND we want to eliminate sending duplicate information, we can solve it the way I did in the pull request. In the pull request I removed the followed_tag_names field and updated our view code to parse the JSON from the followed_tags field. I choose this because it saves us a database hit AND decreases the amount of information we are sending to and from the server.

const followedTagNames = JSON.parse(currentUser.followed_tags).map(t => t.name);
...
<ReadingList
  availableTags={followedTagNames}
  statusView={root.dataset.view}
/>,

TL;DR

By spending an extra couple of minutes looking at your code and thinking about scale, at the very least, you can save your future self some extra work. At the most, you could save your app from a big slow down if/when it gets hit with a large amount of growth.

HAPPY SCALING!

DEV Community: Molly Struve (she/her)

Interview Prep Questions

General Questions

Sr./Staff Level IC Questions

SRE/Reliability IC Questions

Manager Questions

Setting Yourself Up for Interview Success

Interview Often

Flash Cards

Interview Prep Questions

Molly Struve (she/her) ・ Aug 18 '21

Professional Interview Prep

Take Notes

Give yourself 30 min before each interview

Enjoy it!

Good Luck!

Incident Retro: Failing Comment Creation + Erroneous Push Notifications

Context

Problem

Fix

More Problems

Fix 2

Impact

Comment Creation

Push Notifications

Learnings

Large PRs

Missing tests

Technical feature education

Being more intentional

Incident Response

Broken Comment Creation Reporting Flow

Broken Push Notifications Reporting Flow

Technical Remediation

Thanks Everyone 🤗

Goodbye Offline Page

What type of PR is this? (check all applicable)

Description

Follow up

Related Tickets & Documents

QA Instructions, Screenshots, Recordings

UI accessibility concerns?

Added tests?

[Forem core team only] How will this change be communicated?

[optional] What gif best describes this PR or how it makes you feel?

We were wrong

What type of PR is this? (check all applicable)

Description

Related Tickets & Documents

QA Instructions, Screenshots, Recordings

UI accessibility concerns?

Added tests?

[Forem core team only] How will this change be communicated?

[optional] Are there any post deployment tasks we need to perform?

[optional] What gif best describes this PR or how it makes you feel?

Duplicate Digest Email Incident Retro From January

Background Context

What type of PR is this? (check all applicable)

Description

Related Tickets & Documents

QA Instructions, Screenshots, Recordings

UI accessibility concerns?

Added tests?

Added to documentation?

[optional] Are there any post-deployment tasks we need to perform?

[optional] What gif best describes this PR or how it makes you feel?

Bug Trigger*

Mitigation

What type of PR is this? (check all applicable)

Description

Related Tickets & Documents

Root Cause and Learnings

What type of PR is this? (check all applicable)

Description

Added tests?

Handling Hotfixes

TL;DR

Welcoming Katie Davis and Suzanne Aitchison to the Forem Team!

Hi! I'm Joining the Forem Team 🎉

Katie Davis ・ Jan 14 '21