DEV Community: Thai Wood

Use Interview Skills of Accident Investigators to Learn More and Get Better Answers

Thai Wood — Fri, 19 Oct 2018 00:00:00 +0000

Learning from others’ experience is a critical skill. Typically we do this by asking questions. But it turns out that how we ask questions, and how we interact with the person we’re asking questions of matters, a lot.

Cognitive interviewing is a collection of techniques that help people express what they experienced and observed. These techniques, sometimes by other names, are used by a number of professionals in fields where witness or participant recall of information and accuracy is especially important. This includes police departments, emergency medical professionals, and accident investigators.

The cognitive interview technique is one in which the idea is to help the person being interviewed (their person with the knowledge), realize they have that knowledge and help them express it.

When to use this

I find it’s helpful in a training setting, a learning setting, where you’re trying to transfer knowledge or where you sense there’s a knowledge gap. Experts don’t typically recognize their expertise, and often find it difficult to recall what it was like to be a beginner in a given domain and can often unintentionally skip a lot (some of this is because they don’t know what’s important to you).

At the core you need to change how you ask questions to get better answers. If you’ve been on IRC for any length of time, or browsed StackOverflow, this isn’t news to you.

For our purposes we can think of cognitive interviewing techniques simply as guides for our behavior, which in turn can make us more effective when learning from others. These skills are not magical, but they are special. They help us bypass a very natural inclination to give short, less thorough answers.

This process can be thought of as almost a negotiation, they don’t know what you don’t know, and you don’t know what they know, as you proceed you eventually meet in the middle forming a bit of an equilibrium.

An advantage to the cognitive interview technique is that it doesn’t have especially rigid structures, primarily you just keep in mind the objective. This can make it seem more ephemeral and harder to learn for some though.

First off, you should know that I wasn’t formally trained in these techniques. What I’ve supplied below is a cross section of skills I learned in the field in conjunction with research I learned from later. This combination has proven fairly effective for me, and I hope it can do the same for you.

This is not an exact process for the simple reason that people are not exact processes. Knowing more about peers and being able to activate our empathy here is key.

An example

Have you ever had the experience where someone looked at a log file and said something like “Oh, it’s just that process crashing, just restart ?" and of course, wanting to learn, you'd ask, "How did you know that?" or "Where does it say that?" and often the reply will be "It just does" or "It says so".

Many times, it doesn’t really read that way. The person who has identified the problem, has used potentially years of skills and expertise to distill the complexities of the problem to a solution.

They may not even be aware they’re doing it.

Note here that I call these people experts, that doesn’t imply that you dear reader are not an expert, simply that this person happens to be an expert in this particular thing.

By applying some of the techniques we’ll talk about here, you help them verbalize that knowledge and help transfer it.

There is no 1 total or complete “Cognitive Interview,” there are several techniques. This means they can be used as it fits the situation or time constraint.

It can be tempting to think that you as the explorer, since you are seeking, are the one that needs to do the cognitive work, after all, you’re “digging” right?

This isn’t really the case. The person with the first hand knowledge, the person who was there during the event, the explainer, is the one who should be doing most of the mental work during you questions.

Your mental work as the questioner will be to decide which tools in your interviewing/questioning tool box are appropriate to use based on the situation and your evolving knowledge of the person. This is another place where can activate our empathy.

So let’s break down some of these techniques:

Uninterrupted recall

It turns out that when most people ask questions in an information gathering/ interview context that they tend to interrupt a lot. Usually with questions. It’s not that they’re trying to be rude, it’s typically that they’re focusing on guiding the process to get their questions answered. Guiding the process by applying the techniques here is the right idea, but doing it by interspersing more questions, actually makes the whole thing less effective.

Since the person who observed, or even participated in the event has the first hand knowledge, they’re the ones that should be using more of their cognitive function.

So this technique boils down to, don’t interrupt them when asking about an event. Research indicates that interrupting, while not only rude, can prevent the person being interviewed from actively participating in answering the question. When this happens you’re at risk that they’ll simply answer your question in short form instead of expanding on it and giving you the information you might really need to know.

So by not interrupting they’re likely to tell you a larger number of more accurate things!

Encourage them to tell you everything

This one might seem a lot like the last one, but often times the people you’re asking questions of won’t know all the details of what would help you. They have their own expertise, different from yours, and neither of you may be especially aware of that expertise.

To help prevent a situation where the person answering your questions glosses over things you might want to know and to prevent you for having to enumerate everything that you might want to know (which you may not even yet know, encourage the person to tell you everything.

How to do that? Simply ask them. You typically already know your colleagues so adjust this to fit you and them, but something as simple as “Please tell me everything you can about the process, even if you think it’s unimportant”

Tailor the questions

You don’t need to be a robot that asks the same questions of all people each time you use these techniques. Again, use what you know of your colleagues, activate your empathy. Where possible try to tailor which questions you ask to suite the individual.

This is a much more difficult technique, but the more you know about the person you’re interviewing the easier this will be.

In research this often gets called “witness compatible questioning” but it simply means, people are different, they respond differently to different questions, try and keep that in mind.

Encourage “I don’t know”

You want accurate information typically when you’re performing this process. You want recall, no fabrication. It’s not that your colleagues are purposefully making things up to deceive you, but there can be a lot of social pressure to “be the expert” or have all the answers.

At the same time, having someone guess the reason for something, typically won’t do you any good. Research has shown that guessing can occur in questioning any time, even if someone wasn’t encouraged to guess, so encourage them to say “I don’t know” if they don’t know.

“Its OK to not know all the answers here, just let me know if you don’t know something, so I know where to do more research” for example.

You did it!

That’s it! These are some of the core cognitive interviewing skills. There are more, which we’ll talk about in a future installment, but these are the ones that interviewers depend on most often in their toolbox.

If you want to learn it the way the NTSB does, they actually offer a class of their own.

Not sure how to use these? Want to practice? Shoot me an email and I’ll help where I can.

References

Koriat, A. and Goldsmith, M. (1996) ‘Monitoring and control processes in thestrategic regulation of memory accuracy’, Psychological Review, 103: 490–517

“Cognitive Interviewing for Accident Investigators - IM401S.” National Transportation Safety Board (NTSB), www.ntsb.gov/Training\_Center/Pages/2018/IM401S.aspx.

Fisher, Ronald P., Stephen J. Ross, and Brian S. Cahill. “Interviewing witnesses and victims.” Forensic psychology in context: Nordic and international approaches (2010): 56-74.

What do you think? For more like this see here: https://thaiwood.io/DevTo

What is MTTF, MTTR, MTTD, or MTBF? An introduction to incident and service metrics

Thai Wood — Tue, 28 Aug 2018 00:00:00 +0000

In addition to the typical metrics that you may think of as being part of a service, that is CPU, instance count, disk, etc… there is another class of metrics data that tells you about the potential reliability of your service.

These are, MTTF, MTTR, MTTD, and MTBF. These are Mean Time To Failure, Mean Time To Resolve, Mean Time To Detection, and MTBF.

These are all metrics that cannot be observed directly. That is you cannot take a single data point on a graph and say this is our MTTF. That’s because it takes at least two data points and must be computed.

Further, you need to decide on what timeline you’ll compute this. Say over the last year? Six months?

You may have seen a variety of acronyms associated with these metrics, here are some that you’ll encounter:

MTTF - Mean Time To Failure. This is the average of how long between when something goes down. Since its of course up in between failures, this is often just “uptime” averaged over a period. From reliability engineering, this is intended to be used for systems and components that can’t be repaired and instead or just replaced.

MTTR - Mean Time To Repair. This is the average of how long it takes for things to come back up once they are down. This time period represents all the work of repairing the component of the system.

MTTD - Mean Time To Detection. This is the average of how long it takes to realize something is down. So for example if something went down at 1200, but no one noticed or was alerted until 1210, then the time to detection was 10 minutes. If you had multiple incidents over time, you could use the data points to average this.

MTBF - Mean Time Between Failure. Similar to MTTF, but for repairable items.

A warning about incident metrics

I primarily include these definitions so that you can be aware of what they are. It can be helpful/important to know about the existence of these metrics as you’ll often hear their use encouraged.

It’s also important to know that by using these metrics, you can make yourself blind to some more important things.

Most of these metrics come from reliability engineering, but not software engineering. That means the physical world. Even there, it can be argued that many of these metrics aren’t appropriate. If one motor started rusting and lead to failure, would you expect others? Well, it depends on the conditions doesn’t it?

When we talk about people and their behavior in complex situations such as incidents and outages, these metrics become less and less relevant.

Putting too much effort or thought into these metrics whispers the lie that all incidents are the same and if you can control some of these factors than you can improve your incident response.

The problem is this isn’t true. At the very least it’s backwards. Fixing many other things may help these metrics improve. At the worst, focusing on these will keep you from ever asking the right questions, and keep you from getting the right answers.

So how do you begin improving the things that drive these metrics?

Ask questions
Understand that these metrics will never tell you the truth.

You can lay groundwork in a similar manner as other disaster planning, things happen that you don’t expect. All you can do is be well prepared for that.

Plan for what to do when a team member doesn’t know.
Plan for what to do when things are unknowable.
Give your team outlets to talk to you about the process.

Focus on things you can control, like how soon you can detect an incident. Then ask questions about that number.

Questions you might wish to answer/know about your incidents and your team:

Is this an incident type we’ve seen before?
Is this an incident type no one has seen before?
Were docs available for this type of outage?
- Did those docs clearly outline correct action?
How was the incident responder feeling?
- Overworked?
- Underslept?
- Is the first incident they’ve dealt with today/tonight?
- The 50th?
Did the incident responder have the resources they needed and feeling that they could use them?
- You may be surprised to learn that simply saying “you can do this” such as “you can escalate” or “you can restart a service” often isn’t enough.
- Especially if they’ve been yelled at before or the culture makes them hesitant to pull that lever

What do you think? Leave a Comment. Click here if you want to see more like this: https://thaiwood.io/DevTo

Technical Debt Isn’t Like Financial Debt

Thai Wood — Mon, 09 Jul 2018 00:00:00 +0000

I’ve spoken to some leaders at some organizations lately and I’ve been hearing a common lament. Somewhere in discussing their architecture or processes, they look down, shake their head a bit and say something like “Well, we have technical debt,” before considering what solutions are available to them.

Thinking about rework that may be needed in your software and processes before introducing new features is a good idea, but the metaphor of technical debt like financial debt can cause some problems.

Thinking of technical debt like actual, financial debt may make sense on the surface. They’re both “debt,” right?

The problem is there are a lot of false assumptions that can get carried over when thinking of technical improvements you’d like to make as “debt”

Why’s that?

Some traits of financial debt:

All debt gets paid off
Debt has interest rates, making it easily measured and payoff predictable. (Because its interest rate is part of the debt agreement)
You can borrow as long as your credit score holds up
Debt that you ignore will grow
Debt can be used to buy things
You can declare bankruptcy

Do you see the problem here? Carrying around this metaphor often means subscribing to these ideas, many of which aren’t true as it applies to technical problems and solutions and rework.

Let’s tackle the biggest fallacy that results from carrying around this metaphor: That debt must be paid off.

This simply isn’t true. Not every slightly imperfect design decision or code created needs to be revisited and returned to a perfect state or otherwise reworked. Perhaps the bit of code is not read often and runs well. Perhaps it changes rarely.

The opposite can also be true, something small can cause your team to spend a lot of time on it, either through running, patching, or understanding.

Calling all bits of your codebase or process that may need rework “technical debt” can also encourage one to engage in point 3; continuing to borrow, a “put it on my tab” mentality, where once you have debt, it can be even harder to be aware of the problems caused by adding more.

Some of these traits do overlap, you can ignore code or make design decisions that will cause rework later in exchange for shipping. This is sort of “buying” something using your “debt” but the price isn’t very clear, making this a potentially difficult trade off to evaluate well.

You also have the ability to declare bankruptcy. You could just say “we’re over it, we’re doing a full rewrite,” which somewhat frees you from the previous debt, but also places a burden on you to build something new. You’re not resetting to zero, you’re actually going negative.

Am I saying don’t “pay off” any “technical debt” or that you can never do a rewrite? No, of course not. I’m saying that thinking of it like debt at all, as something you “have to” or “should” pay off without thinking further on it and being more intentional is suboptimal which can keep you from missing the point.

This doesn’t mean you can’t use the metaphor or need to remove it from your vocabulary. But instead, that there can be benefit from considering and conceptualizing the problem a bit more directly, as a burden, instead of an often rarely measured, poorly defined pool that you can continue to dump things into.

“How much human intervention will be needed because of this [change, thing, whatever]?”
- This helps filter out what is just technically perfect, e.g. an older technique framework, etc.. from what will actually cause someone to have to respond, repair, or otherwise react.
“Does this keep us from improving elsewhere?”
“Am I concerned about something concrete or is this premature optimization?”
“Are many people going to get tripped up here and/or sink a lot of time because of this?”

Do you disagree, have you found the financial metaphor helpful? Are you “paying off” the debt for the sake of it? What is your experience in balancing shipping vs rework? Shoot me an email

Confidence is a Key Metric

Thai Wood — Wed, 27 Jun 2018 00:00:00 +0000

Whether you’re the CTO, an engineering manager, a team lead, or an individual contributor; when you’re building and developing systems and process, whether for release or otherwise, a key metric to keep in mind is confidence.

What does that mean? That means that you need to build a system that is trusted. For example a CI system that let through every build wouldn’t be very confidence inspiring, even if some users might see that as “broken,” how could you ever trust the system to catch your mistakes or to keep your product from going off the rails?

What happens to systems that don’t inspire confidence? Eventually, nothing. That’s the problem with systems that don’t instill confidence; more often than not, their use will decline until they’re not used at all.

In the absence of a trusted process, individuals will seek to replace that system with something they can be more confident in.

These are often “hope driven” systems.

Hope driven systems are usually less productive, like manually checking and rechecking the commits in a release, or reading and rereading a patch and hoping a potential bug will jump out the 5th time that didn’t pop up on the 4th (or 3rd, or 2nd). Or with lack of confidence in a result (say a test run or pipeline flow), retrying with the hope that the result will be different or more revealing.

Does this mean double checking things are bad? No, not at all. Simply that ad hoc “security blanket” type systems that replace humans with computers instead of the reverse for things computers are good at like checking tests and confirming outcomes is usually inefficient.

Certainly more inefficient than rolling out to a single box in prod and giving 1% of traffic for example. Something that a system may make possible in a matter of minutes as opposed to the hours or even days that panicked reading and rereading takes.

If your system isn’t producing confidence, then its likely being subverted or causing time to be wasted.

They should produce more than the end result. They should produce knowledge and confidence in the process itself and in the operators’ skill along the way. It doesn’t have to be a lot at once. I’m not saying that someone should run through a checklist or other process once and feel like they’ve mastered it, but over time this should be noticeable effect.

Here are some things that erode confidence:

Intermittent outcomes
Random failures
Unreliability.

Things that instill and improve confidence:

Accurate, helpful error messages
Date of last accuracy check

Note that not all of these need be applied in all situations.

What processes or systems do you have in place on your team that build or destroy confidence?

What do you think? Leave a comment here. For more like this see here: https://thaiwood.io/DevTo

Your First Step in Automation

Thai Wood — Tue, 13 Feb 2018 02:16:00 +0000

Your first step in automation

First write it down. The first step to move you to automation is to first write it down. You may think that you don’t have a process today that you can automate. But unless today is your first day in business you already have some process ready that you’re using. If you have code, an app, or a website that is in production that customers see, you have a deployment process. You have a deployment pipeline whether you realize it or not. Set aside 30 minutes to go through this activity. Depending on the complexity of your product and your (or your team’s’) familiarity with the process it could be longer or shorter, but 30 minutes is usually enough to give you an idea. If your deployment process is currently powered by a post it notes, you still have a process. You can move from no automation to simple automation. All you have to do is write it down. Don’t be embarrassed and think that you can’t have automation because your process is a mess. Don’t think, “I’m not there yet”. The way you get there is by starting now. What do you write down? You start by asking yourself some questions, and writing down the answers. What would happen if everything went perfectly? Just write that down. It might look something like: first a developer will push some code into a repository. What happens after that code is in the codebase? Just focus on the ideal path, don’t worry at the point about anything other than that. Does someone copy it to a test environment? Do they build it? Write that down. What happens when tests in the test environment succeeds? Does it go to a new environment? Write that down. Then does it get tested again? Write that down. Once it succeeds there what happens? Does it finally go to production? Perhaps the way it reaches production is by it getting copied to a server and a service getting restarted. Now that you have all of the steps written down go back through the list. And just note what actual action is associated with each of these steps. You may have to ask your team about this. Is the copying done by first putting your as ssh key somewhere? Write that down.

Do you have to copy it into a box or other secure environment first? Write that down. Once it’s up and running with the new code, how do you know it’s OK? Is there a graph that you check? Do you just refresh the web page many times? Is there a health endpoint you visit? Whatever it is, you guessed it, just write that down. Now let’s take a look at that list from the top. There are going to be steps where multiple actions could happen. At first we focused on everything going right, but now write down what happens if it doesn’t go well. What happens if the tests don’t pass? Do you file a ticket? Do you open an issue? What about when it reaches production and it doesn’t look good, the refresh doesn’t work, the graph isn’t happy or the health check is bad? Do you copy an old copy of the binary? Do you reboot first? Do you read logs? Whatever it is, just write that down.

Got that list? Okay good. You now have a solid foundation for building automation and reducing toil so that your team can focus on delivering features.

This list you’ve created isn’t just an intermediary step, it isn’t just a throwaway document. You now have a foundational document for onboarding new team members. So you have automation.

What do you think? Leave a comment! For more like this see here: https://thaiwood.io/DevTo

Do you use version control?

Thai Wood — Tue, 13 Feb 2018 02:13:00 +0000

You’ve probably heard it before, “You should use version control”. This one is a critical. In many cases other recommendations can depend on the size of your team, your organization, its goals, or its values.

Not this. This is a bedrock foundational recommendation. If you want to achieve any sort of reasonable, helpful automation, you must use version control. Why is this? Because you have to be able to recreate previous states if needed. You need to be able to rollback. Your team needs to be able to look back, to see what’s changed. To see how they got to the state today.

If you took your car to a mechanic, he’d ask about what happened before your car got there. How do you drive? What did you experience? What changed? Imagine going and just saying each time “I have no idea, I spaced out every time I drove, and never listened to the noise of the engine.” That’s what it’s like without version control. You can’t answer basic questions.

This isn’t just for when things go wrong though. Your team can be use this information when developing features. They’d be able to check if the feature or a similar one existed before, but was perhaps removed, for example.

It’s a good base for new team members to review as well. It can help them understand how features tend to get developed. That is, fast or slow, many at once, or just one at a time. They can also get a feel for the code review process (or start code review in the first place) and team style this way. Understanding what gets asked about or ultimately revised can help them integrate into the team faster.

Version control also lets you test different versions of your app. Your team can figure out how various features or specific implementations stand up to benchmarks or to your test suite.

Even if you don’t feel that your team is large enough now to warrant this, starting the process from the get go will make it easier to expand the team or get help later and is much safer than allowing potential features to languish on individual developer machines.

Finally, it’s important to actually use version control, not just have it available somewhere. If your team doesn’t use it, it allows the possibility of code and knowledge getting lost and won’t help you.

What do you think? Leave a comment. For more like this see here: https://thaiwood.io/DevTo

Do you have a rollback plan?

Thai Wood — Tue, 13 Feb 2018 02:12:00 +0000

It’s true that DevOps practices and CI can make your deploys much safer. That doesn’t negate the need for a rollback plan though.

A rollback plan is exactly what it sounds like. It’s a list of steps you’d take to undo a release and restore the system to its original state.

You or your team might think that you have one if you have an engineer who can undo things, but that isn’t the same as a rollback plan. Having a single or even a small fraction of the engineering team know how to undo a given release isn’t a replacement for a rollback plan.

A rollback plan is a set of written steps that others can follow to undo the release.

Why have a way to undo a release if you have the other pillars of automation (code review, automated tests, etc…)? Just like you wouldn’t want to get placed into the middle of the woods without knowing how you got there and how to get home, you wouldn’t stage a release without knowing what steps would get you back.

Writing a rollback plan can also help clarify what impact the release is expected to have on other systems and what other steps should be taken.

Would you have to restart a service? Dependents of that service need to be notified.

Would you have to make a change to a database? If its shared with other applications those owners should be notified.

Would you have to restore data? Better make sure the backups are working and restorable.

What does a rollback plan look like?

Here is a sample of what one might look like for an organization that has a release pipeline and both automated and manual tests. Yes, its entirely possible that yours is longer, but this is an example of where you can start. If you already have something like this, you can improve it by making sure you have a link to a document that answers the question “how?” for each of the below:

Release v5 of SuperImportantApp Enterprise Edition:

Scheduled: 1/2/2018 0700 PST

Rollback steps: Should the deploy fail for any reason, the following steps will need to be performed in order: Set CI to retrieve release tag 4.5 artifacts Deploy resultant artifacts to QA Confirm automated tests pass Release artifacts to Prod Rolling restart of service on all nodes 5a. See our internal document hub “Rolling release” document for how to do this Confirm tests pass, both manual and automated

What do you think? Leave a comment. For more like this see here: https://thaiwood.io/DevTo

Do you have a record of what the right configuration is?

Thai Wood — Mon, 12 Feb 2018 16:22:00 +0000

Unless you’re just starting out, you already have servers doing what they need to be doing. They got configured and built in a datacenter or launched in a cloud. Either way, there was some amount of configuration that helped shape how it does its work. Just because you or your team configured it once, doesn’t mean that it’s recorded at all or even correctly. The core question to ask of yourself and your team is: “Do you have all the information you need to create an identical version of that server in a minimal amount of time if needed?” This could be part of disaster recovery. Something catastrophic happens and you need to make a new server. Do you know what that would entail? Or it could be part of something good, maybe you’re growing and want a new app server, or something more powerful. How do you make sure that the new server has all the features, configurations, and functionality the old one did?

If you don’t have the answers to these questions, don’t worry, it’s not too late to gather them.

Some questions to ask as you go through recovering or recreating this information: What software must be running on the machine for it to do its job? What configuration needs to be present for that software to work the way you need it to? Are there special users that need to be created on the machine? For example, for automation or support Are there limits or quotas on a the machine that need to be put in place or removed? What operating system does it need? Is there a specific version or patch level? What sort of network connectivity and firewall rules does the application need to be present or absent?

Once you have this information in hand, not only are you on your way to having a more robust disaster recovery process, you’ll also have begun to pave your way to have infrastructure as code and the automation that allows.

With this information at the ready you’ll be able to respond more quickly to disasters, limiting your downtime and the time that recovery takes. You’ll have opened the doors to automating your infrastructure itself.

Further, you’ll be able to use this information to answer questions and solve problems. Perhaps you’re subject to an auditing standard. With all your configuration on record and accessible, you can begin to answer questions like “Are we using a secure version of our operating system?”, “Are our firewall rules too permissive or strict?”

Already did this? Great! The next step is to automate this configuration and provisioning, using infrastructure as code.

What do you think? Leave a comment. To see more like this click here: https://thaiwood.io/DevTo

Beginning DevOps Tools

Thai Wood — Mon, 12 Feb 2018 15:43:00 +0000

Even though DevOps is a cultural shift, not a job title, there are still tools that can help break down silos and tools that are associated with the movement. This means that if you’re applying for a “DevOps job,” you’ll likely be working with certain tools. This is why I use the word in the title of my book This is also where beginners tend to be drawn to DevOps, at the tooling level. Really, this isn’t so bad. “How can I use the tools that organizations who move quickly do?,” is a pretty good question as long as one isn’t blind to the idea, that mental models and culture can play a huge role as well. So with that in mind, what tools should you learn if you want to be “DevOps”? There are no exact tools, it depends on your organization, its wants, needs, and goals. That’s DevOps for you. But there do tend to be some categories that these tools fall into. And like programming languages, once you know one, it’s a lot easier to learn another since many of the concepts you already know will map.

Categories are:

Configuration Management
Source Control
Communication
CI/CD
Monitoring

For configuration management there are a few options:

Chef
Puppet
Salt
Ansible

Do you need to learn all of these? Probably not, and certainly not all at once. It’s better to learn how these tools work in general, what paradigms do they follow, etc… and then pick one and learn more about it.

For example, Salt follows a Master/Minion (client/server). Chef follows a similar model. But Ansible doesn’t have a centralized server.

For source control, though others are around, I’m just going to say learn git unless your organization has another requirement for you. If you or your team want a place to learn and try it out, you can use Try Git

For CI/CD, its important to think about what you’re trying to accomplish, but many tools can be used to accomplish similar goals. Don’t let your team get bogged down in the weeds, its more important to get a basic pipeline up and running and improve from there. Options include:

Jenkins
Drone
Travis
Gitlab

You’ll notice I also listed communication. This might not seem like a DevOps or SRE thing at first, but really communication is what helps you break down silos. It also can help you resolve an incident faster. This could include tools like chatbots, or even your system for contacting relevant people like PagerDuty.

For monitoring Just as with some of the above tools, it’s more important to consider what you want to monitor and to what extent than it is to worry about a specific tool at this point. If you’re going to monitor some metric, you should be able to answer some questions about it. First, what does this data tell me? If it doesn’t tell a story of some sort, its highly likely it shouldn’t be monitored or you need to investigate what the data means to your organization. Secondly, to determine if you should be alerting around that metric, “What do I want someone (possibly me) to do in response to this metric?” also “when?” and “how quickly?”

If you there is no action to be taken around a metric, it’s not a good candidate for alerting.

What do you think? Leave a comment. To see more like this click here: https://thaiwood.io/DevTo

What is infrastructure as code and why should you use it?

Thai Wood — Tue, 12 Dec 2017 00:00:00 +0000

Infrastructure of code is just the process of writing down what makes a particular machine unique, in a way that both computers and humans can understand. (When I say infrastructure here, this is most often the servers that make up and support a given application or service.)

Depending on what configuration management software you choose to help you with this, you might be using Ruby, Python, or even Bash.

Does it run an http server? Does it have an increased number of file descriptors? What config files need to be in place for it to operate?

This resultant code can be run to recreate a machine or if it was changed incorrectly, even bring it back inline with how it should be if changes were made manually.

Having the definition be code allows you to do some things like:

Versioning it. You can know when a change was made, by whom, and see the difference between versions.
You can review changes. Since the definition is code you can begin to apply your peer review process (you do have a peer review process right?).
It makes it easier to do the right thing. Ever find yourself SSH’d into a server (perhaps even prod?!) and making a change, all the while thinking “I’ll document this later, after I make sure it works.” IAC helps you avoid this. The way to make the change is the same way to codify it.

How does this differ from documentation? The code is human readable, but also machine runnable, so servers can be recreated/updated as needed where as with docs, there are separate steps, update the machine (by hand?) and then update the docs.

With infrastructure as code, defining/updating the state that you want is the same thing as documenting it.

Infrastructure as code is not the same as documentation. Just having the definition of a server codified and recreatable is not a replacement for documentation, but it allows you to focus on your docs on different things.

Instead of having to go through and write everything down all over again about what’s on the server, you can write about why that server does what it does and how it relates to other infrastructure or applications.

For example instead of having to say Server X is a CentOS server that runs Application Y and has ruby version 2.2.3 and is firewalled with such and such iptables rules. You can instead focus on docs like server X is a server for Application Y and is critical for it to continue running. If you need to know what’s on the box and its version, see the code.

What do you think? Leave a comment. For more like this see: https://thaiwood.io/DevTo

Bootstrapping a Node With No Internet

Thai Wood — Sun, 13 Aug 2017 00:00:00 +0000

When you bootstrap a node using knife bootstrap Chef assumes that you’ll have access to the internet. It uses this to download the client package and some metadata, but you don’t have to be connected to the internet to bootstrap a node.

This is especially handy in cases where you have a firewalled setup that won’t let you get packages from the internet.

There’s two ways to solve this. The first is to take advantage of the --bootstrap-install-command flag.

For distro’s that use `apt`

$ knife bootstrap chefnode -N MyNewNode --bootstrap-install-command "curl http://your-internal-server/chef.deb -o /tmp/chef.deb && dpkg -i /tmp/chef.deb

For distro’s that use `yum`

$ knife bootstrap chefnode -N MyNewNode --bootstrap-install-command "yum install -y http://your-internal-server/chef.rpm"

This method is a good choice for one-offs, or a very small number of machines, but if you have anymore than that then the better option is to make a bootstrap template.

Making your own bootstrap template

Bootstrap templates are simply erb files that Chef uses to determine how to bootstrap a node. You can override the default one with the knife flag --bootstrap-template

You can make your own template and place it in ~/chef-repo/.chef/bootstrap (you may have to make the bootstrap directory).

The easiest way to do make your own template, is to start with the default Chef template and modify it to contain the bootstrap commands you need, similar to the above.

Once you’ve created and saved your own template, you can now change your command to (assuming you made a debian.erb):

$ knife bootstrap chefnode -N MyNewNode --bootstrap-template debian

You may have noticed that these two methods are functionally the same, pass the command in and it gets interpolated into the template, make a template and put your command in, same result. The reason I recommend you use templates for more than a few nodes is because you can keep your bootstrap files in version control, though it takes a few more steps.

Version controlling your bootstrap files

Since knife will look in a few places for a .chef/bootstrap directory, we have to keep our bootstrap files there somehow. The problem is we shouldn’t commit .chef directories to version control since the directory contains keys. Instead what you can do is make a ~/chef-repo/bootstrap folder that contains your files and instead of creating the directory as we did above, instead we’d symlink it.

From your ~/chef-repo:

$ ln -s ../bootstrap .chef/bootstrap

Now you can add your ~/chef-repo/bootstrap directory to your next commit without exposing keys or having to keep track of a bunch of bootstrap commands.

What do you think? Leave a comment. Click here if you would like to see more like this: https://ThaiWood.IO/DevTo