DEV Community: T.J. Telan

How to Build a Custom Integration Test Harness in Rust

T.J. Telan — Mon, 26 Apr 2021 00:00:00 +0000

I ran into a problem effectively using cargo test in Fluvio for integration testing

Let’s talk about integration testing in Rust

While creating integration testing for Fluvio, I ran into a problem. Organizing and executing integration tests with cargo test was becoming inefficient. We needed to standardize the setup of a test environment.

As a lone developer, you can apply one-off customizations when running tests locally. But if you try to extend that strategy to continuous integration, you’ll quickly find that making changes manually becomes burdensome. CI encourages testing many different configurations, which means a successful CI plan requires easy management of test harness variables (a.k.a. Not manually updating variables for every test you need to run).

cargo test is just not equipped to handle this specialized focus on environment setup, or the cleanup/teardown needed after a test is run. When using cargo test, these crucial tasks could only occur outside of the harness or within the logic of a test. Neither of these are good choices. Outside of the harness is not ideal because these processes end up too disconnected and hard to maintain. Likewise, including setup/teardown within the logic of a test is inappropriate because it creates mental overhead for a test writer, and may obscure the results of tests.

I needed to find a way around the limited functionality of cargo test -- keep reading to find out how I did it by creating a standardized setup and teardown as part of our testing harness.

How does `cargo test` work by default?

There is a distinction between unit tests and integration tests in the Rust book. The distinction is less about testing strategy and more about defining Rust’s conventions for test organization.

The main points are that:

Your tests are annotated with #[test]
libtest harness enumerates through all of your tests (a point we’ll revisit later in more detail)
libtest returns the pass/fail status of the execution

What do I need from integration testing?

Libtest doesn't specifically offer anything to support integration testing patterns.

Setup of a standard test environment – especially in a complex system – is essential for managing expected behavior when making code changes.

Unfortunately libtest does not assist with setup or teardown. I needed the ability to abstract away the setup and teardown of my test environment from test code.

This task will be performed either way. Without harness support, setup/teardown will be performed via external shell scripts or padding the setup/teardown process within every single integration test... (no one's idea of fun).

It isn’t convenient to manage setup and teardown in a different context than the integration test. This kind of testing overhead leads to hard-to-reproduce and time consuming mistakes.

Where do we get started with a custom test harness?

By default, libtest will compile each of your #[test] labeled functions into their own binary crates (with its own main()) and executes it as part of the test. But we’re going to build all our integration tests into a single crate. This is recommended in order to speed up compile time ([1], [2])

First we’re going to create an integration test directory at the root of the crate where we’re going to build our integration test focused binary.

$ mkdir integration
$ touch integration/main.rs

# Then create a main() function in main.rs

In your Cargo.toml, you want to add

# Cargo.toml

# We'll revisit the `inventory` crate later in the post
[dev-dependencies]
inventory = "0.1"

[[test]]
name = "integration"
path = "integration/main.rs"
harness = false

This tells cargo test to not use libtest when running the integration test.

When we run cargo test integration, what cargo will compile integration/main.rs and execute it in the same manner as cargo run. This is all a harness is from cargo’s perspective.

Add Setup and teardown steps

Next we’ll lay the foundation for our testing pattern. We’ll create 2 functions, setup() and teardown(), and add them to our main() (with reserved space in between for our future tests to be called).

// main.rs

fn setup() {
   println!("Setup")
}

fn teardown() {
   println!("Teardown")
}
fn main() {
   // Setup test environment
   setup();

   // TODO: Run the test

   // Teardown test environment
   teardown();
}

Collect all integration tests

To do its job, our test runner needs to create a list of all the test functions. Initially, I thought there would be an easy way to do this by leveraging libtest's #[test] attribute.

I dug around in relevant areas of libtest and Cargo test and Rustc macros code, but long (sad) story short, there is no straightforward way to reuse libtest for the purpose of test collection.

If that surprises you, then you're like me. I had hoped to use the test collection functionality from #[test], but it wasn’t clear how I could accomplish this. My mental model for how cargo test works needed a refresh.

Now that we’ve removed the option of using libtest, so that gives you 2 practical options for collecting tests:

Manually modify integration/main.rs and add your test in between the setup and teardown
- A quick and straightforward solution if you have a small set of tests
- This option requires us to add new tests to this list, which can be error-prone and tedious as we grow.
Build a test collector. We generate an external test catalog, and modify integration/main.rs to execute tests from the catalog.
- This is a long term solution, which we’ll be covering for the rest of the post.

Building the test collector

For this test collector, we'll be utilizing a crate. The inventory crate is a plugin registry system. We'll be using it for all the heavy lifting in our test framework, which means we'll be treating our tests as plugins.

In our main.rs, let’s declare a new module tests, where we can organize all the integration tests.

// main.rs

+ pub mod tests;

fn setup() {
   println!("Setup")
}

fn teardown() {
   println!("Teardown")
}
fn main() {
   // Setup test environment
   setup();

   // TODO: Run the test

   // Teardown test environment
   teardown();
}

In our new module, we’ll start by creating a struct to represent a single test for the plugin registry.

// tests/mod.rs

#[derive(Debug)]
pub struct IntegrationTest {
   pub name: &'static str,
   pub test_fn: fn(),
}

inventory::collect!(IntegrationTest);

In this example, our struct IntegrationTest has 2 fields.

name is a human-readable name, which can be used as a key for test selection.
test_fn is a pointer to a function whose signature is non-async, takes no args, and does not return anything.

Note: You can use functions that take args, and return things.

For example:

pub test_fn: fn(String) -> Result<(), ()>,

Then we call the inventory::collect!()macro to instantiate a plugin registry. When we write our tests, we’ll add to the plugin registry. More on this next.

Adding new tests to plugin registry

We’re going to add a new basic test to the plugin registry. Start by adding a new submodule called basic in the tests module.

// tests/mod.rs

pub mod basic;

In the basic module, we write our basic test basic_test()

// tests/basic.rs

use super::IntegrationTest;

fn basic_test() {
   println!("Running basic test")
}

inventory::submit!(IntegrationTest {
   name: "basic",
   test_fn: basic_test
});

We'll use inventory::submit!() to register our new test with the IntegrationTest struct we defined earlier.

name is a friendly, human-readable name. We can use this name as a key to search through the plugin registry.

test_fn takes the name of our test function. It has the same function signature as we defined.

Running tests from registry

We’ll finish this example up by running all of our registered tests

// main.rs

pub mod tests;
+ use tests::IntegrationTest;

fn setup() {
   println!("Setup")
}

fn teardown() {
   println!("Teardown")
}
fn main() {
   setup();

-   // TODO: Run the test
+   // Run the tests
+   for t in inventory::iter::<IntegrationTest> {
+       (t.test_fn)()
+   }

   teardown();
}

$ cargo test integration
   Compiling blog-post-example v0.1.0 (/home/telant/Documents/blog-post-example)
    Finished test [unoptimized + debuginfo] target(s) in 0.21s
    Running target/debug/deps/blog_post_example-e042d787684bb333

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

    Running target/debug/deps/integration-7ed2452642c6f3b6

Setup
Running basic test
Teardown

Tips for extending the example

The example runs all of the registered tests. But here are some useful impls if you want to extend even further. For example, adding a CLI, if you want to select individual tests. Or provide options to customize setup or teardown behavior.

impl IntegrationTest {
   pub fn all_test_names() -> Vec<&'static str> {
       inventory::iter::<IntegrationTest>
           .into_iter()
           .map(|x| x.name)
           .collect::<Vec<&str>>()
   }

   pub fn from_name<S: AsRef<str>>(test_name: S) -> Option<&'static IntegrationTest> {
       inventory::iter::<IntegrationTest>
           .into_iter()
           .find(|t| t.name == test_name.as_ref())
   }
}

If you want to see more of these ideas extended even further, check out Fluvio’s integration test runner.

We use the CLI to customize setup, handle async testing, and we use an [attribute macro] to collect tests.

[attribute macro]: https://github.com/infinyon/fluvio/blob/master/tests/runner/src/fluvio-integration-derive/src/lib.rs

Conclusion

Rust’s testing ecosystem in the 2018 edition is great for unit testing. But for integration testing it still has room for improvement. Custom harnesses will become more necessary as Rust gains new developers.

If we want to avoid reinventing the wheel, we need stable support from libtest or more examples of how to perform test collection and patterns for setup, test, and teardown workflows.

If you made it this far, thank you for following along with me! I wrote this because I could not find a guide to do this before trying to do this myself, and knowing these things beforehand would have made it much faster. Hopefully others find my experience helpful.

9 Insights I Uncovered While Building a Writing Habit

T.J. Telan — Fri, 29 Jan 2021 00:00:00 +0000

Building a writing habit

In the forever-trying year of 2020, I set my sights on earning the 16 week posting streak badge on Dev.to – and I did it! (Then I took a long break from posting.) Before I get started on more tech writing for 2021, I wanted to stop and reflect on this effort.

How did it go?

Pretty well, to be honest. It has been one of the most challenging goals I’ve set for myself. After the 50% mark, I had exhausted the developed ideas in my backlog so it became harder to get in front of my schedule. I had to start the next post immediately after publishing.

I found a few insights while building my habit for writing, and you might find them helpful, too.

1. The hardest part is getting started

It’s easy to say that some of this is due to the mood of 2020, but it was really easy to fall into self-doubt and self-rejection based on the worry that no one would read what I was going to write. Or that people would read it, but they’d hate it.

These are still feelings that I sometimes have today. But aside from SEO, marketing on social media, experimenting with the time I post, and other tasks that take effort – whether or not someone clicks and reads is out of my control.

Due to the mental effort required to overcome these feelings, the hardest part of writing is getting started. As such, it (almost) doesn’t matter what the resulting quality is. Primarily what matters is getting something on one topic on the page, to edit later.

2. Editing is easier than writing

I have a much easier time editing than writing from scratch. Knowing this about myself meant that it was a lot more important to complete any kind of first draft, no matter how bad.

The editing phase takes at least twice as long as writing. This phase may involve complete rewrites of sections, but I still consider that to be less difficult than starting from a blank page. It’s also a lot easier to judge how much I have left to do when the edits are focused on formatting and pacing, and less on content or coherency.

3. Don’t be a perfectionist

At least for the content that I’m creating, most people aren’t going to suddenly understand a complicated concept because I wrote a specific sentence in a particular way.

For the goal I was working toward, it was important to post on a predictable schedule. While it’s good to set your own standards for yourself high, keep in mind that it’s impossible to level up 10x with a single post. Writing well takes practice, and you get more practice if you are able to let go, call a story done, and move on to writing something else.

4. Writing consistently doesn’t always yield consistent quality

I’ll be the first to admit that I think some of my posts are more useful or interesting to read than others. That’s just how it goes sometimes, where ambitions are better than the content produced.

Ira Glass spoke about storytelling and the taste gap (and how to close it). He encourages that the most important thing one can do when starting out is to produce quantity over quality. Over time you improve, if you can endure this creative, iterative phase. It is normal for quality to lag behind your ambitions.

With that in mind, it’s okay to write things that you never publish. I’ve learned that writing can be a method for understanding a topic, or just organizing my thoughts. Not all of that needs to be public.

What continued to give me encouragement was noticing that my first draft quality was (on average) getting better over time. That was a measurable improvement!

5. Be really specific about who you are talking to

When I write, I always start the first draft always with a question: Who am I writing to?

It doesn’t always matter if the answer is a real person. This person I’m writing to might be based on my past experience, questions people have asked me, interactions I’ve observed, an organization, or just completely made up!

What is most important is that I focus my message so my intended target audience understands. I don’t want to unintentionally expand the audience or over-explain a concept if it doesn’t benefit the base group of people I am writing to. This technique has made it a lot easier to write in bursts, and edit for clarity. It helps each piece of writing have a consistent voice.

6. Writing titles is hard (because SEO is hard)

I have to admit: I change the title of each post no less than 3 times, sometimes minutes before publishing the post!

Different publishing platforms have different audiences, so experiment with what works for your readers. If you plan on cross-posting across multiple platforms or you want to be discovered through search engines, you will need to learn a little bit about marketing and SEO.

I’m frequently torn between using technical jargon in my titles, or less specific concepts that can be generally applied.

Playing with this balance is a cheap and interesting game that, in my experience, avoids disappointment by taking a long-term perspective. It’s been validating to see what kind of content does well though organic search visits vs. people sharing links. I’m still caught off guard when an article from many months ago begins to trend. The lesson here is to be patient and see what strategies pay off.

7. Keep a backlog of ideas

I have a spreadsheet that I use to keep track of ideas in various states. When I think of something that might be a good topic to explore in a post, I add it to the list. These can be title ideas, posts that are in progress, and links to drafts.

I try not to dismiss an idea too early. Just writing it down gets it out of my mind, with low pressure to fully explore the idea. Sometimes this collection of thoughts ends up presenting patterns or themes that, when combined, contain enough material to create a full post around. This strategy is really practical, and comes in handy when I am feeling like I’m in a creative rut. If I have a deadline to meet, I can always pull from the pile of ideas I already had.

8. Building structure and motivation for writing is moving target

I am generally informed about trending topics within tech, but I’m trying to discover my niche and I don’t want to spend an eternity looking for ideas. I’ve also discovered that I’m most creative when I have constraints and have a structure to lean on.

I have a long-term goal to work for myself creating educational products / services / consulting. So a structure I lean into is that each post contributes to that long-term goal.

It’s also important to address shorter-term motivation. My approach has been to identify what motivates me to create, and then slowly build systems to support producing content. Aside from the obvious need to set aside time to write, creating that motivation for a post takes three things:

The content has to be something that is not too difficult for me to create. This means I mostly know the material I’m talking about, and I’m not learning new skills specifically for the post. (It’s not uncommon for me to write about a learning experience, though! I just don’t set out to learn how to do something specifically to write a post)
Something I enjoy creating - The topic I’m writing about needs to be interesting, or the story I’m telling needs to be meaningful. Otherwise, I’ll get bored writing it (or you’ll get bored reading it).
The result should provide value to someone else. People value their time, so in writing an article, I’m trading my time writing for someone’s time savings. What I communicate can save people time, and that’s meaningful to me. I know I have benefitted from others doing the same.

I really enjoy creating software developer advocacy and tech education content, and the topics I (mostly) write on reinforce these areas.

I aspire to have objectives that can be described as SMART. Admittedly, my short- and long-term goals are still pretty abstract. I think that’s okay right now, as long as I am making progress in the right direction.

9. Don’t be afraid to be seen trying

Credit to a video by Evelyn From The Internets for helping me articulate my feelings about this point.

This relates to my previous point of starting being the hardest part. Some of my internal friction stems from the feeling of being afraid to be seen trying something I’m not good at.

With some rare exceptions, I’ve decided to keep my old content online. It likely won’t be an issue where stuff I created in the past while I was still learning has embarrassed me. Largely, this content is ignored and no one but myself cares about it.

To be concise, haters gonna hate. That’s what they do. So don’t pay any attention to the inconsiderate folks who “just want to help” but are not actually helpful. The tech community is full of strong personalities that aren’t always inviting or pleasant to engage with. (I’m not saying this is the majority. I see you...)

I see enough 💩 being talked in social media threads that it constantly occupies space in my mind when I’m about to publish a new piece of content. However, the truth is that most people are not focused on you for even a moment. So a way I like to view that is that most people will not even see you fail – that’s the blessing of no one knowing who you are yet. Therefore, there’s no reason to not try.

Conclusion

Building a writing habit has been very humbling. I learned that I actually know a lot less about certain topics than I thought. I had to research more than I expected. But as I gained experience, I leveled up my time management and my process.

Now that I’m not trying to post weekly, what I would do differently is to spend more time developing my production processes. And that’s what the focus of this next year will be about.

Aside from continuing to stream my product development on Twitch, this year I will be creating short videos, starting with some of my more popular blog posts as topics. I hope this will lead into more types of educational content, and more opportunities to work with other people in achieving their goals.

I started a consulting business (Figbee Labs) in the latter half of 2020. I will continue to travel out of my comfort zone into the deeper waters and hopefully meet new people or organizations that want to work together. It’s important to me to help people from all backgrounds succeed in sustainably building and running their own software products. I will document my experience over 2021 through posts and various social media.

If you’ve undertaken a similar challenge, what did you learn about your process? What do you want to achieve with writing this year?

Questions to Find the Right Continuous Integration System for You

T.J. Telan — Thu, 10 Dec 2020 00:00:00 +0000

Continuous integration is fancy shell scripting. It even uses shell scripting terminology like pipelines, but is obscured with marketing and many seemingly complex and different methods of configuration.

The purpose of this post is to discuss Continuous Integration (CI) Systems at a high level if you are considering switching to a new service. I want to introduce you to a few talking points that aren't discussed enough in order to help you make a decision.

What are the goals of Continuous Integration (CI)?

The functions of continuous integration are very flexible and therefore, and the goals are super simple. Automatically run scripts upon the occurrence of an event.

Common events can be a git commit or a timer event set to run every 5 minutes.

How does CI differ from shell scripting?

Continuous integration has a narrow context in which it is intended to run. It has extra structure in order to tell a tighter story. (The CI story leads into another story called Continuous Delivery/Deployment (CD), but we will only be focusing on CI)

A best practice of CI is to automatically build and/or run tests on every commit in order to validate whether the codebase builds/tests are complete without errors.

Basic structure defines conditions as well as commands or actions that should be executed when an event occurs. Conditions such as a new commit in a specific branch, or file paths. Or keywords in commit messages.

On the more advanced side, platform specific features like plugins may be introduced for sending or receiving events from other systems, initiating external triggers based on other pipelines, caching, cleaning up and notifications.

What type of CI systems that exist today?

There is no shortage of software that is designed for Continuous Integration. What I believe sets these platforms apart from each other is:

The experience and effort involved with setting up a new environment for your organization
Path to configuring new pipelines, and maintaining them over time.

If your organization regularly exercises the creation of new environments, like in consulting orgs, it is reasonable to gain the experience with an open source tool that won’t cost you any money with licensing.

Otherwise, I might suggest picking a system the reduces the friction for pipeline creation and maintenance.

Styles of setting up new pipelines falls into two categories.

Stateful (WebUI) pipeline config

Jenkins is the primary example of configuration via WebUI, as it is the functionality out of the box. Adding pipelines in this system requires a user to log into the WebUI, configure connections to builder systems and methodically define pipelines.

Pros

The biggest player in this space, Jenkins, is free to install with flexible system requirements.
Relatively low skill required to get started. Setup of pipelines and any subsequent edits are performed through the browser with a few clicks.
Many plugins exist, which probably suit the needs of your org.
Most of the CI in this category are older and mature. Very search engine friendly solutions.

Cons

Administration is more difficult and time consuming to perform as footprint increases. Over time the details of the pipeline configuration may drift and be difficult to reproduce for the purposes of documentation or migration.
It is easy to neglect this kind of CI installation. Both a testament to stability, and a detriment to your org when unexpected outages occur. Rebuilding can be tedious and error prone.
Takes a lot of effort to secure natively in a cloud-based future. Typically these CI systems inconveniently end up hiding behind a VPN.

Stateless (File-based) pipeline config

This is the modern style of CI system. Your TravisCI, Github Action, Azure DevOps

Typically we find this system is hosted through your git provider, or as a separate hosted service that we allow access to the repos. YAML is the most popular config format.

Pros

Configuration of pipelines stay versioned alongside the codebases that use them.

Cons

No config schema standards, though. most use a yaml format.
Complexity becomes harder to manage, and requires a commit and push to repos in order to test. Testing contributes clutter in git logs, and uses up build minutes, which is the cost unit that many hosted services bill on.
Limited self-hosted options that are free.
Limited self-hosted options that have small system requirements

My opinion

I strongly prefer the stateless type of configuration because it provides a more code-like arena where documentation can be provided.

Mixing the workflow of CI pipeline configuration with your organization’s version control practices can be disruptive if you need to push code in order to see the results of changes to your pipeline. It is a price to pay for the ability to easily correlate changes in codebase with changes in pipeline.

The balance to strike rests on your code hosting provider needs (i.e. costs per user) and the types of technologies and platforms you need to support for building (web, native, mobile, embedded, etc). This flexibility mostly skews support towards building for web-based technologies.

Also, Jenkins technically has file-based pipeline support via Jenkinsfile, but unless my org was already deeply invested in staying with Jenkins, I would steer clear. It is not the future. It is harder than necessary to use.

What are the CI Hosting Options?

Self-hosting

Self-hosting means running and maintaining a process on a physical or virtual machine that you have access to.

Pros

Somewhat more private for less effort if hosting in a private location.
Effective use of data center computers. Initial costs may be higher, but maintenance costs are typically lower.

Cons

Being responsible for maintenance
Limited to the platforms that you have available.

System as a Service (Hosted)

External systems as a service (Hosted services) allow users to just log in and connect their version control.

Pros

Not responsible for performing maintenance.
Typically authentication is provided, with options to support your org’s auth provider

Cons

Can be expensive if you require a lot of build time or resources
Shrinking caps on “free” resources for open source projects

Issues to Consider when Switching between CI platforms

These are some categorized rhetorical questions that you or your organization should ask if you are migrating CI platforms

Threat model

Do you need to be deployed on-site?
What are the upfront and ongoing costs associated?
Do you have secrets (like private keys, API tokens) that are needed at build time?

Contributor model

How are contributors logging in?
What are the costs associated per seat?

Maintenance

Who is responsible for the health of the pipeline?
How much time/money are you budgeting to spend on maintenance and improvements?

Cost of Build minutes

For open source projects, how many build minutes are being used?

Responsiveness needs

How many concurrent builds does your org require?

Technologies used

Containerizing builds is popular for isolation. But not all projects can build within a container.
Apple, Microsoft and embedded projects are not nearly as flexible and may require special considerations or resources.

Delivery/Deployment model

What do you need to happen after a build passes?
Are you creating a build artifact that needs to be stored somewhere?
What is the desired frequency between build vs delivery/deploy?

Conclusion

Not all continuous integration is considered equal. But they all provide basically the same functions. In most cases, it is possible to self-host and do everything yourself. But sometimes it is worth paying money to a product / service that takes care of your needs. It is really all about making sure you know what the trade-offs are for your specific case.

Re: Reading More Technical Content Can Help You Improve Your Content Creation Skills

T.J. Telan — Thu, 03 Dec 2020 00:00:00 +0000

This post is in response to Stephanie Morillo’s post

The main piece of advice that is elaborated upon is to read a lot of technical content.

What I think is very helpful about this post are the questions that are asked. I thought it would be a good idea to answer these questions so that I can share my opinions around how I read through technical content.

Why do I like a piece of technical content?

I can answer this in different ways depending on the context that leads me to find technical content. The main two contexts being discovered through social media, or via a search engine.

Through social media

I spend a lot of time on technical social media, like Reddit, Hacker News, Dev.to, and Twitter and I mostly only read the trending posts. So in these cases, an individual’s or company’s personality drives my interest over the actual content.

A lot of the time this leads to some cult of personality for popular posts in the comment section. This can’t be helped. However, this kind of engagement occasionally leads to interesting and tangential discussion. I like when the author can participate in the discussion as well, but this is a rare occurrence.

I prefer concepts over code in the case of social media discovery. Emphasis when I’m reading through something that is about an ecosystem I don’t have experience using. I like having my hand held lightly, and guided through the examples with metaphor or images.

Bonus points if you both provide links for more context, as well as provide a simplified summary. Help frame how I should think, and even give negative examples for how I should not think if it helps.

These posts are primarily to entertain me, and then to inform me. I want the attention to readability with layout and markup to be more considered. If there are images to help set tone, I’d rather they be well used memes, or diagrams.

Through search

If I’m searching for something technical, it’s more likely that I’m doing research, design or implementation. So personality may continue to play a role in my likelihood to revisit an article, but the code snippets and coherence of an explanation is more important.

Bonus points if consideration was made for keyword searching or organized textbook-like. I don’t need my hand held as much in his case. Just point me in a direction. I don’t want the entire post to be read through carefully unless the post is about low-level detail.

I’ll be grumpy, but I will endure a post that is harder to skim if the content is promising enough.

Who is their intended audience?

Part of reading a lot of technical content means reading things that are not specifically targeted at myself.

When the content involves discussion of code, I really prefer a lot of gentle, but explicit nudging around the skill level required to make sense of what is in the post.

Wide audience is generally my least favorite because content has to be high-level enough to not lose anyone. If you’re aiming wide, then say so or be very obvious.

Textbook-style is good enough for me. Build up concepts in a reasonable sequence. Or link to posts or definitions that talk about foundational concepts that you’re introducing. Unless you’ve already stated an expected skill level, I will feel like you don’t know the audience you’re writing to.

What does this (company/individual) do really well?

This kind of question is mostly for judging a collection of posts. Any single post may have left positive impressions on me of being informative, concise, funny, easy to read. But I don’t feel like I can judge what the competencies or patterns of a company or individual is until I see other work.

I look for consistency in voice. Are we having a conversation. Am I reading a monologue? Do you know where your content beats and break things up with whitespace or images? Are you flexible enough with how you present ideas so that I understand you or know what homework I have in order to understand? Visual branding also comes into mind when discussing consistency.

What does this content help me do?

My expectations of what the content offers me is again, dependent on the context in which I was introduced.

In general, if it is through social media, I’m looking first to have some kind of entertainment aspect. I’m looking for a breadth of information that allows me to insert my own ideas. I know that content creators aren’t always in control of where and when their content is posted, so this isn’t always going to be fair to judge entertainment value when the content is informational. But it is a plus if the author can be engaging and educational because it takes effort to do well.

Conversely, if it is through organic search, I’m looking for some kind of informational value. Help me solve a problem, or understand a model of thinking better.

How would doing (or implementing) “X” improve my content?

This is a great question. Often when I ask myself this question, I’m considering the visual branding of an individual. But lately I’ve also been thinking about how my favorite creators structure and pace the flow of information in their posts while balancing their voice. This is why it is a good idea to follow content creators that resonate with you, because everyone learns to strike this balance differently. And typically the more someone writes technical content, the more natural and easier to read they get.

What can they improve upon to make this content even more effective?

At least for me, it is easier to edit someone else’s work. Naturally, I’m constantly thinking about how little bits of friction can be smoothed out and so that I could understand it better. This could mean mixing the styles of creators together, and I try to identify creators who are particularly good (or sometimes bad) at one thing so I can pay attention to how their style evolves.

Conclusion

I don’t know if this post will be useful to others, but it was a good exercise for me. If you’re a content creator, I encourage you to read through Stephanie’s original post. Ask yourself these questions and try to give concrete answers.

Quick Primer to Practical Reproducible Builds: Reference Environments

T.J. Telan — Fri, 27 Nov 2020 00:00:00 +0000

Overview

This post is inspired by a larger Twitter thread by James Munns about industry practices to follow for embedded systems projects that are not covered in school. Particularly the concept of a reference environment. However this idea is important across all platforms of software development.

Our aim is to be pragmatic to a wide audience of “Developers” and not to be confused with the goals of the reproducible builds project. (I’m not worrying about achieving complete build environment determinism. There is a lot more involved that is out of scope for this discussion, but I encourage others to check it out.)

What’s the benefit?

The benefit of a reference environment is to allow a practical step-by-step plan for individuals to reproduce a suitable environment in order to build from a codebase, produce artifacts and interface with any external hardware.

I want to help you form a maintainable process of documenting what is needed to build your code, and create artifacts for release.

The expected outcome is to create a reference environment for build and release purposes.

An additional side-effect of this effort is a shortening of the effort to onboard new devs and testers into new projects or codebases. Additionally version control of this process will reduce your organization’s institutional knowledge.

Your reference environment can be used for

CI environments
Onboarding new developers
Projects that don’t get modified very often
Embedded projects
Mobile projects
Projects with private code
Projects with vendored / version pinned code
Legacy code

Start your reference environment in steps

First level: Automate what is easy and Document what is hard

You need to have a step-by-step set of commands to run that walk you through downloading and installing your tools in order to enable others to build their own reference environment. Tools and libraries installed through a package manager, and anything manually installed off a website.

The goal of documenting the setup of your local environment can then be further streamlined the more you automate the step-by-step into scripts that you can provide.

The experience you want to cultivate is a relatively short step from a fresh clone, to building and modifying code.

If you use a language that has its own package manager used to compile, such as Javascript’s NPM, Java’s Maven or Rust’s Cargo, then you are able to keep track of the names and version numbers of the libraries you use.

I recommend that you learn how to use these tools to also remove files created during build. Cached files from previous builds are classic causes of “works on my machine” and may hide otherwise obvious issues. Make a habit of trying to reproduce issues starting from a cache-free build by using the same tools for build to also remove the files and directories it creates.

Not all language package managers come with cleanup functionality, so you may need to install plugins. Otherwise you should use a tool like make, and create separate make build and make clean targets.

Second level: Shell scripts

Automation should be driven by makefiles or plain shell scripts like bash/zsh/fish/sh or in windows batch or powershell. The rationale is that these are plain commands that you can copy/paste from your terminal, and into the script. You can always call out to another language like python/ruby/perl or other tools.

But the initial point of functionality should be plain shell scripting through an executable shell script or a Makefile.

If you are working from an existing step-by-step document, then you should have a sequence of commands that you can transfer straight into your script. This is a lot fewer commands for your future users (and future you) to deal with.

Third level: Configuration management

There are tools that specifically deal with configuration management of your operating system / development environment. One of their main selling points is declaring the desired state through config files instead of scripting everything step-by-step. The tools read the config files and check the system for differences. If differences are found, then these tools take specific actions in order to attempt to bring the reality of the system state to be the same as the config.

The config files lend themselves into achieving automation that is “idempotent”. A simple explanation of idempotency with respect to configuration automation is experienced when running the automation twice in succession. The first time, an automation tool takes a moment to execute because work is being performed. The second time, the tool discovers that there is no work to be done so the execution time is significantly reduced.

Examples:

Ansible
Saltstack
Puppet
Chef

In your shell scripts, you need to install these config management tools first. But then the tools can replace significant volumes of shell scripts in many cases because their idempotent properties lead to more efficient execution. The more popular tools have an added benefit of wide support of different OSes, allowing your configs to also widely support different OSes.

Next level: Portable Reference Environments via Virtual Machines & Containers

Building your reference environment within a virtual machine may enable you to be a little less strict about documenting and automating your setup. After you build the environment, you can save a snapshot, back it up or even distribute it to others. This can be convenient and portable in most cases.

You can also use your scripts or config management to configure virtual machines, or containers. Using tools like Vagrant or Docker allow you to store the instructions for building virtual machines (via Vagrantfile) or containers (via Dockerfile) alongside your code. This enables potential contribution without the need of installing tools locally because the codebase includes a portable development environment.

Platform specific caveats

Virtualization or containerization is not an available option for every platform. In these cases, you may be limited to config management or documentation for your reference environments.

If your project has the following, you may have limited support for a portable reference environment.

Apple products
Platform-specific tools like IDEs and/or compilers
Hardware requirements such as GPU
Very old “Legacy” OSes

Conclusion

The specific tools, and libraries you use in development may seem obvious to you. But as a courtesy to an outsider, or even to yourself in the future, you need to provide requirements needed to build your code.

Documentation requires discipline to keep up-to-date. Scripts, config management or even a portable environment such as a virtual machine or a container are even better because they are easier to verify for correctness.

The less critically someone needs to think in order to get started, the easier it will be to focus on what matters. Reference environments are not one-size-fits all. They may be unique per project, or organization. Do future you or future team members a favor and write these details down. It can save a lot of time and avoid works-on-my-machine.

Rust in 2021

T.J. Telan — Tue, 17 Nov 2020 00:00:00 +0000

I missed the official call for blog post submissions, I still want to share some ramblings of my hopes for Rust in 2021.

I’ll split this up into two sections: What I hope to see from the community, and what I hope to see for myself.

What I want for the Rust community in 2021

I want to see Rust shed some of its reputation for being hard to learn

When I was learning Rust, I thought it was harder than necessary. But I would admit that it was due to writing strict idiomatic Rust while I was still figuring out how to be productive.

I think learners should give themselves patience to learn Rust by writing bad or mediocre code first, and focus productivity later.

I am not exactly sure how to create the conditions for this outcome. Maybe more people can publish more Rust that looks messy but “just works”.

I know that Rust in Action is being written and it is for an intermediate-level audience. But perhaps this looks more like encouragement to beginners to ignore the warnings from the compiler. Just write Rust code and form your own opinions.

More Rust in DevOps tooling

In my experience, the cultures surrounding Rust and DevOps are very similar. They both seem to attract cult-like appeal that demand adopters to adjust to thinking in a different way than they may have been taught.

The intended reasons are for a “better future experience” that usually alludes to improved cooperation at a non-technical level.

The implementations of Rust or DevOps shift considerations of future problems earlier in development. Possibly experiencing some of the friction earlier.

Rust’s benefits are often placed in a context of pure development benefit, which may be ok for now given it is primarily Devs that are writing code. But as a user, there are a few tools in the space that I’m aware of that support DevOps culture (e.g. Habitat, Linkerd) and I want to see more of them.

Link regarding rise of Rust + DevOps: https://d2iq.com/blog/rust-devops

Better Rust Interoperability between languages

Rust is entering other ecosystems through libraries via FFI. People don’t even have to know or care if they are using Rust.

However, if I’m writing the code myself, it would be nice if FFI were done without needing to be so mindful of C as the common language.

C++ and Java come to mind as currently Rust interop is a complicated dance. Rust to Javascript is improving through the Deno runtime, or through WebAssembly.

My motivation is that I’d like to work in the ecosystems where these languages are most often used (Web, Mobile and Embedded), but I stubbornly want to use Rust as much as possible.

More shared experiences from people picking up Rust as a second language.

The borrow checker has a reputation that scares people away.

Especially if they are not from a traditional CS/Engineering background. And we should be trying to bring these folks into the ecosystem.

Acceptance of Rust as another language worth knowing by “average devs” would be helped with more beginners writing or talking about it. It’s my observation that a lot of developers who are self-taught or from bootcamps are mostly in the Javascript/Python/Ruby space for mostly Web facing code. But Rust can be a perfect second language to learn about types and memory management.

I believe content from this crowd would be my favorites to read since I think they’ll give valid opinions to someone who has been using Rust for years (like me) now overlook or accept without second thoughts.

Increase of copy/paste templates to help bootstrap common types of projects

I have just become aware of cargo generate because of some of the niche project spaces I’m working on: WebAssembly and embedded.

Today, I keep my own repo where I offer rough working examples using common servers or patterns.

My technical wishlist

Rust analyzer support for generated code
- I want support for Tonic’s generated code
Better tools or patterns for debugging Async/Await
- The debugger just makes me realize how little I understand about Futures.
Native Rust for building games
- The end goal being Rust on consoles or in AAA games. I know they are sometimes used through FFI. It’s not a light task to use Rust in this manner.
Native Rust for building mobile apps
- We can technically write apps for both Android and iOS today, but they aren’t exactly popular because building is complicated

Where I plan to be with Rust in 2021

I plan to write more Rust that other people find useful.

I hope to have OrbitalCI into more of a Beta quality, and usable by others for feedback
Publish more crates that allow people to write Rust to interact with existing APIs

I plan to have serious projects

A few serious projects in upcoming Rust niches so that I can produce more educational content (and hopefully even a product that others may use!)

Embedded
WebAssembly
Mobile

I plan to start making money with Rust.

I’ve started my own business in 2020, and I’m not yet making any money. However, I’ve decided to base a lot of my work using Rust. I am enjoying the experience so far. I intend to sharpen my skills throughout 2021 into paid work.
I hope to make some income through Twitch from livestreaming my coding and other platforms from my writing. (edit: Added my affiliate links 😉)

How do I intend to help the Rust project?

I’m using Rust to build software for my business (in the DevOps tooling space).
I will be producing more blog posts, with an angle of inviting Python and Javascript programmers to Rust
I plan on making my first code or docs contribution to the Rust project in 2021.

Conclusion

I see a lot of promise in Rust's future in 2021. Much has become stable in 2020 and all that is needed is some time and encouragement.

I have enough confidence that I plan on using Rust as my main language as well as motivation to get more involved with the community.

Is DevOps Automation a zero-sum game?

T.J. Telan — Tue, 10 Nov 2020 22:09:03 +0000

Some of my intention is to lean into the ambiguity of the definition of DevOps.

In general, when automation is created and introduced into an organization for the purposes of "progress" for a wide audience, is this improvement gained at the expense of an individual or a team?

Here's a made up example. Automatic alerts in a production environment of a web app for a small US company's online order form.

The primary customer live in US timezones. The assumed culture is that outages are to be avoided. 24/7 uptime.

Before alerting was introduced in this org, outages in production might not be known until traditional work hours when more complex workflows.

After alerting was introduced, the possibility of non-work hour wake up calls from alerts increased significantly. On-call/event-based compensation might balance the fairness to the person responsible, but that's a monetary cost paid from the org to offset a forced work-life cost.

In my opinion, the introduction of automation led to compromise by the individual, with practically no benefit to the business to make no change.

DevOps Helps you be a Better Leader

T.J. Telan — Fri, 06 Nov 2020 00:00:00 +0000

This is the first of a series about helping your organization embrace DevOps.

DevOps is misunderstood

DevOps is buzzwordy – so hot right now. Like all tech development trends, everyone wants it, but everyone also has their own ideas about the kind of problems it will solve. This is a precarious situation, because a lot of people aren’t wrong. But they are unaware that their version of the truth is so narrow that they end up missing the big picture.

In this post, I’ll discuss DevOps at a high level by addressing:

What is (and what isn’t) DevOps
How DevOps differs from the “normal” development process you may already have
How practicing DevOps can reduce time and money required for development and operations
How nurturing a DevOps culture will increase the quality of life for everyone working on your product

There are many more topics to discuss, but we must stay focused! This should lay the foundation to allow for constructive conversation about how DevOps works in your organization.

Who should read this post?

This post is primarily intended for people who need to know the purpose of DevOps because they’re managing a team or hiring for a DevOps skill set. You also might find it helpful if you’re just wondering what DevOps is all about as a developer, operator, student, or in a related, non-technical role.

Conventions and Definitions

Let’s start with some definitions and conventions so we’re on the same page.

Environment or space may be used interchangeably. It refers to the location where code is being executed. A laptop or workstation are examples of environments that we can physically touch. But Kubernetes, or “the cloud” are examples of more abstract environments.

Development environment will refer to any environment that is primarily used by a single person for the purposes of writing code, such as a laptop or a workstation.

Production environment will refer to any environment that is shared, regardless of whether external customers are using it. An internally used environment used only for testing purposes is a production environment. Big differences between a development environment and a production environment may be complexity or scale at which components are deployed. As well as the amount of targeted automation to perform changes such as deployment or collection of analytics.

Development (Dev) is a phase and a team we associate with writing code. Specifically, it refers to code that has not yet been released officially to a wider audience. When a project is in development, there is an implication that it may not yet be stable. The people who do the code writing are called Developers, or Dev.

Operations (Ops) is a phase and a team we associate with building and maintaining the environment(s) used for running code from Development, as well as infrastructure used to support the running code. The people who run code may also be Developers, but they are also commonly known as Operators, Ops, SysAdmins, SREs, Production Engineers, and increasingly (and incorrectly) as DevOps Engineers.

DevOps is the art of joining principles from traditional software Development and Operations so both can work better together. DevOps is not a phase of development, nor is it a role, despite the emergence of DevOps engineering roles or teams. It is an organization’s professional culture, and emotional intelligence for working together as a single team. The purpose of DevOps is managing expected behavior of the product in production with respect to constant “change of state” (we’ll talk about that soon).

The problem DevOps solves

An industry trope is the antagonistic rivalry about whose role, Dev or Ops, is more important. But it doesn’t need to be a battle. DevOps is a culture or method of interaction between Dev and Ops and a direct result is shortened feedback loops between writing code (development) and running code (operations).

Admittedly, “DevOps” is a more complex concept when compared to Dev or Ops, which muddies an otherwise straightforward conversation because the actual tasks of practicing DevOps are not one-size-fits all.

Software is rarely a single moving piece. The benefit is the flexibility to incrementally change the state of any single piece. The “change of state” refers to updates to code or the supporting infrastructure, and effectively communicating those changes to the people involved. As a system grows in complexity, it takes some effort to make changes without disrupting other pieces.

DevOps, as a practice, is about managing the changes to those moving pieces in an organized and intentional manner. An example of where DevOps builds this bridge is deploying to production. Thin encourages releasing and running code in a way that Devs and Ops can anticipate each other’s changes, at a high rate. These procedures are preferably automated, for speed and consistency.

Change the job title on your reqs: DevOps is not a role!

Unfortunately, DevOps is increasingly used as a catch-all for all engineering tasks that do not have an owner. The result is that it starts to lose any specific meaning.

A recent (and regrettable) trend in job postings is to rebrand SysAdmin or Operations roles into “DevOps engineer” roles. This makes it more difficult for companies and job-seekers alike to make a good match.

Case in point: coding is often listed as a requirement for a “DevOps Engineer” role, but if that role is actually an operations role, there will be few opportunities to write code. This means that new hires joining with the expectation that their role includes coding will feel misled, and turnover will be increased.

What does a solution look like?

So what is DevOps if not a role? Let’s describe practicing DevOps through the actions of the members of the technical organization.

DevOps-enabled Dev vs Pure Development

Unlike pure development, DevOps is primarily about improving processes and analytics capabilities to remove friction from developers’ workflows, outside of writing code. It is not about product functionality or quality of the product code. For example, Devs define the terms of success or failure of the behavior of their code in production. But thanks to DevOps practices, they can analyze that data to inform their next actions.

Devs in a DevOps practice have a shorter onboarding process as well. Common tasks such as software requirements for build code and deploying code are documented through automation that is kept up to date. With this, new devs can focus on the codebase on day one, not spending all day reading documentation.

DevOps-enabled Ops vs Pure Operations

Unlike pure operations, DevOps is proactive about observing how running code is behaving from within the code rather than only observing results. The purpose is to maximize the value of human time by only requiring a person to look into a problem if existing automated strategies fail to resolve it.

Additionally, the introduction of Continuous Integration/Continuous Delivery (or Deployment) pipeline (aka CI/CD) is an important collection of automation shared between Devs and Ops. The Devs know how to push code for build for the CI, and the Ops know how to pull build artifacts for deployment.

DevOps-enabled management

There is a lot to be said about DevOps-enabled management, but I will keep it brief (but stay tuned, this will be the topic of a future post). For now, I will be focusing on communication, quality of life of your employees, and a short aside on saving money on cloud operational expenditure.

As a product manager, DevOps directly serves the priorities and product roadmap, as they will be naturally driven from the analytics of your production environments. More objective discussion can be had with your developers and operators when you can provide more specific details and requirements to what is motivating your decision making as a manager. Data about feature usage or responsiveness can more easily be measured and placed onto a dashboard if the metrics to be tracked are known during development, rather than after a feature is complete. This production data can help facilitate discussions with your stakeholders.

As a hiring manager, DevOps enables you to be more specific about what it is you are looking for in job applicants. If you’re looking for a candidate with broad skills, you can justify why with actual functions that need served. Otherwise, you can also be much more specific with need for specialized roles since the functions of your direct reports feed into the quality of your analytics.

Lastly, if you operate in cloud infrastructure such as AWS or Azure, your DevOps practice should condition you to analyze your compute usage for ways to save money. It can show you when you may be able to shut down compute resources for the night to save costs on guaranteed idle time that you would otherwise be paying for. Automation can automatically turn them back on for the work day if needed by devs, but otherwise, you may find that these resources are often unintentionally abandoned by devs.

Conclusion

If you are a manager, how do you start a DevOps practice? You start by consistently being data-driven through requests for data from your employees. Show them how the data influences your decision making. The consistency of your requests ought to naturally lead to automation of data collection.

Remember that DevOps is more than operations. It is a professional workplace culture that starts with management through commitment to improve communication.

To do both drive and support this, management must be very specific about what information they need from their product, then align their processes to reinforce these needs. Realize that this kind of organization of processes is optimal. The perceived constraints your employees experience provides very clear expectations for team success while allowing individual opportunity to be creative and fulfilled without setting expectations that it is any individual person’s responsibility.

DevOps is a practice that produces solutions that scale to the needs of the organization and humanizes the people who create solutions. It allows people to specialize their craft while providing a structure to let them trust that the rest of their surroundings (infrastructure) will be ready to accept any of their new creations.

Bottle rockets in my DevOps? It’s more likely than you think.

T.J. Telan — Thu, 29 Oct 2020 00:00:00 +0000

Why is DevOps such a hot trend when many find it hard to concretely define? It is just for big companies? Could you be practicing DevOps as a small software organization and not even know it? What are the outputs of DevOps activities? How do you explain DevOps to your less-technical colleagues?

To help address those questions, I want to tell you a story not about software, but instead about bottle rockets. We’ll take a look at typical DevOps activities and how feedback loops can naturally form and provide opportunities for improvement using DevOps thinking.

Imagine you are building a company that makes and shoots off fireworks. You’re just starting out, so you’re the one making the fireworks.

The firework is aimed, a fuse is lit, the rocket launches, and features are displayed to your customers. You intentionally design pyrotechnics with an expected outcome. Your customers love the loop-de-loops, whistling, and colorful sparks!

At first, you’re just building customized fireworks by yourself with cheap kits. Based on positive feedback from customers, you start to develop a specific type of firework.

This is essentially how all software products start.

You soon outgrow the capabilities and customizability offered through the kits, and to save on money you develop your own process using raw materials. You build and package every single unit in your garage, and have fireworks shows in a field next to your house. People are buying tickets to your fireworks shows, regardless of how the product was built -- they just want a cool explosion (and yours are the coolest).

What your audience doesn't see are the many experiments and failures that went into building their experience. However, since you work alone, any mishaps are in your private space and they are easy to put out with a fire extinguisher and a bucket of water.

This is similar to software development.

Your company wants to expand their market size and offerings. You hire new people to help you build more of what currently sells, as well as help create new designs based on feedback from customers. Building in your garage no longer suits the needs of the business.

This is similar to the start of some legends in software development.

You move into a bigger warehouse with an indoor launchpad. You teach the new people how you work. You build and package units by hand. The new people are not as good at this as you are and they don’t have your intuition, so the quality of output is not as consistent.

You’re now launching your bottle rockets from inside the warehouse, through a window. The inconsistencies sometimes cause a small fire so everyone has to stop what they’re doing to help put it out.

You spend the money or time and effort to create special tools so that they don’t needlessly make mistakes.

This is similar to dev tooling. Dev tooling is domain-specific software or automation created for the purpose of helping developers keep their focus on the details of their problem. It also helps developers to offload the mental burden of tedious tasks such as data processing, reformatting, or validation, onto their tools.

Your company now creates many different types of fireworks, and has showcase displays every week. Every product uses specialized parts so it can be a scramble to get everything done in time. One of your new bottle rocket designers discovers that with a few small modifications, many of your products could use the same, adjustable fuse. Now, many firework designs can be made at the same time, and each fuse can be adjusted in the field prior to ignition. You apply this strategy to other parts and this improves your manufacturing efficiency.

This is similar to microservice architecture , a style of service-oriented architecture design that enables an application to be composed of several small modules organized by business capabilities. These modules can be developed and updated ad-hoc without requiring a full redeployment of all components.

You get late notice there’s going to be a huge fireworks expo very soon, and your company’s unique fireworks will be the star of the show.

You return to the product lines to assist with packing units. It’s a terrible experience.

The work is manual, and for some of the tasks you realize that the same tools can be used by an assembly machine and performed with more precision and consistency. With some extra effort, more machines could be constructed to create units for quality control testing, and package units for shipping. The process of assembly, packing, and launch preparation is now more standardized, and your human workers can focus on designing better fireworks instead of remembering the exact order and specifications of assembly.

This is similar to CI/CD (continuous integration / continuous delivery/deployment). CI/CD is a process that connects development (writing the code) to operations (running the code).

Typically initiated by a developer pushing code, which builds, tests. This process is called Continuous Integration (CI). The end of a CI process that results in an artifact ready to be deployed is called Continuous Delivery. Going one step further to an automatic deployment of the artifact into production is called Continuous Deployment.

As your rockets get bigger, you notice that the aiming from the launchpad isn’t going as well as it used to. With the smaller rockets, you were able to eyeball the right trajectory, but this isn’t working as well with these more complex fireworks. Due to your previous standardizations of fuses, the same fuse modules are used for all the rockets. Therefore we can narrow our investigation of the trajectory issues specifically to construction of the complex fireworks. Quality control is now also improved because there are fewer variants introduced. Changes are easier to track and correct using information from internal quality control and from customers.

This is observation through metrics and monitoring. The trajectory was noticed not only holistically because of regular launches internally, and by customers. But also objectively, because we’ve collected data on the distances experienced in a variety of conditions.

You’re building bigger and better bottle rockets, and your customers are thrilled with the exciting displays you’re putting on! They’ve never seen brighter colors or more interesting designs!

Conclusion

Using this somewhat silly story as a stand-in for the software development and deployment process, you can see that DevOps activities are often part of a developing and growing system. With every deployment, you’re trying to shoot a bottle rocket out of the window.

Picking and choosing DevOps activities to optimize is normal for small but growing software organizations. Be pragmatic. Adopt solutions when you have problems they solve.

So how do you know what you actually need? You’re likely already addressing the needs you have (but in a much less efficient way). Interested in investing in your team’s DevOps practice? Here are some starting points to think about:

If your team spends a lot of time moving between projects, consider bulking up your dev tooling.
If you frequently find yourself solving the same problem repeatedly or copy/pasting and lightly modifying service code, investigate implementing microservice architecture to do that job for you.
If you rely on a single developer’s machine or struggle to consistently build or release minor changes, CI/CD could be what you’re missing.
If you make changes based on feelings and not with data, then improve your product observation through collecting metrics and monitoring.

A cohesive strategy for packaging, deployment, and response is essential as growth occurs. As the rate of updates increase, the rate of outage-causing failures climbs proportionally.

It is fatally optimistic to rely on the hope that your product is absent of critical bugs. Reactivity is not a dependable plan, especially in the high-stress situations caused when your bottle rocket explodes on the launchpad. Instead, assume that things will inevitably go wrong sometime, somewhere. Create tangible strategies that bridge writing code (development) and running code (operations) through common language and goals, and rely on procedures that considered the important details back when stress levels were not high.

My Hacktoberfest 2020 is Complete

T.J. Telan — Mon, 19 Oct 2020 00:00:00 +0000

This is a followup post to my previous Hacktoberfest related post.

TL;DR - Officially I’ve completed Hacktoberfest 2020!

The beginning of the month was a bit rough, and while there were good intentions, the level of communication to participants was not up to par. But eventually the news of the changes rolled out in a much more official way! For what it’s worth, I approve.

Better late than never

The change to make eligible repos opt-in will hopefully provide less of an unexpected disruption to open source maintainers. I imagine the spike in activity will belong to repos who want the commits. Also I hope the intended effect will cut down on greedy devs gaming the rules with low quality contributions to get free stuff.

The extended review time was still 2 weeks, but a couple days later after my post, I got an email acknowledging my 4 PRs. I’ve just been waiting for the clock to run out this entire time.

However, the time is here. Officially I’ve completed Hacktoberfest 2020!

Congrats message in my profile

My strategy

If you have never contributed to open source before, I can share what I usually do. I took a simple approach.

I crawled the Github search for Hacktoberfest labeled repos that were written in Rust because I was only interested in writing in that language.

There were quite a lot, so I picked a handful of the smallest Rust projects where the repo owner asked for some very specific contributions that were clear enough that it didn’t require me to ask questions before getting started. All things considered, it only took me a total of 2-3 hours to write code.

I did a bit of overcommunication with the repo maintainer to be friendly, but also to give the impression that changes can still be made with their feedback. I wanted to offer an opportunity for them to give me feedback so they would be happy with what they would get merged in.

If you didn’t know, you can reference issues from your commit message, which I did. Additionally I linked the PR to the issue, and I left a message in the issue itself describing my thought process. There were small changes requested by the maintainers, though every PR was eventually accepted and merged.

The Prize

Am I a digital lorax? Because I wrote code for the trees

So I’ve decided to plant a tree instead of receiving a shirt. It was nice that there was an alternative.

There was only one color and one size. Fun!

I don’t know what’s next. After completing the form I was told to await instructions from Digital Ocean for how to continue the process of planting the tree.

Either way, this is the end of the line. Hacktoberfest 2020 is done for me. I feel happy to have made a positive impact.

Thanks to Digital Ocean, Intel, Dev.to and the Hacktoberfest team for making this year happen!

In the future I will likely crawl the Github search for small Rust projects to contribute to live on my Twitch stream. If that interests you, or if you’d like me to contribute to your repo, then I’d appreciate it if you’d give me a follow!

What can I do to feel like I belong in the Tech community?

T.J. Telan — Tue, 13 Oct 2020 00:00:00 +0000

When I was a teenager, in the 00’s, the internet gave me net-positive energy and optimism. It was still very novel, and there was a lot of opportunity to discover niche communities that overlapped with my interests. I felt like part of an in-crowd and it deeply influenced my life.

I have continuously been connected online (Metaphorically. Bc I grew up with dial-up). Much of the novelty has long worn off.

It’s been an exhausting year

I was laid off early in the year due to the pandemic. Due to the sudden loss of human interaction, I have been revisiting some of that yearning for connection as I transition to starting a business from scratch (more about that some other time). As a necessity, I am dependent on the internet for the majority of my human interaction. Maybe this is me being mentally old, but I feel the current energy of my echochamber to be more net-negative and I’m hoping I can introduce myself to people who build others up.

Niche communities are difficult for me to discover with the same ease as when I was young. That’s not a reason enough for me to stop looking! So that’s fine. I’ll start looking where I’m at. For professional reasons, I need to start in Tech. So I’ll need to be more specific, because Tech is HUGE.

Searching for my k-nearest neighbors

I’ll spare you the rest of the thought exercise. I’ve been focusing my efforts within the Rust community. Why? It’s reignited my joy in writing code again, and I’m trying to start my business with it. And because people love it!

With a few years under my belt, I’ve finally reached a point in my Rust development that I no longer consider myself a beginner and I can be productive. Great! Well, now what?

What have I been doing?

Regularly posting blog posts with a purpose of educating or sharing an experience as a non-expert
Hanging out around Tech Twitter and occasionally commenting and retweeting
Streaming writing Rust code for a few hours at least once a week
Joining so many Slack/Discord/Matrix/Gitter/IRC communities. Mostly lurking, but participating occasionally
Maintaining a Rust crate and occasionally asking related projects if they would like to depend on it

What have I been expecting?

Part of me was hoping eventually after seeing the same same faces, same names… that I would feel like I had joined the community and I’d stop feeling so intimidated. Perhaps that I would have experienced some subtle mutual acknowledgement of acquaintance? (I have since been lowering my expectations, fwiw. I now believe it is my responsibility to initiate the interactions with potential personal meaning or expectation.)

What have I not been doing?

Actively inviting conversation
Stepping out of my comfort zone to ask for feedback

Won’t you be my neighbor?

When was the last time you joined a new community? What was the defining moment when you felt like you belonged?

I don’t feel unwelcome in Tech, but I feel invisible. My attempts to extend my reach out to those who might actually care to connect has random-like results. Not for lack of trying. But admittedly, I am placing too much value in establishing consistency before figuring out what works for me.

I don't know what options to try next. I’m not fishing for pity, but I am looking for advice. How do I find my audience? How can my audience find me?

A first-timer's perspective to Digital Ocean's Hacktoberfest 2020

T.J. Telan — Mon, 05 Oct 2020 00:00:00 +0000

So... what is Hacktoberfest?

Digital Ocean’s Hacktoberfest is a month-long event over the month of October aimed at generating an increase of contributions to open source projects. Participants are asked to open PRs against public repositories in Github with “positive contributions”. Swag is offered to the first 70k participants to open 4 qualified PRs.

A typical hacker

And… how is it going?

I have mixed opinions that honestly, skew negatively. But I feel like I’m more charitable about my feedback than what I've seen on the internet. I’m not intending to drag Hacktoberfest but I’d like to see improvement.

2020 is my first year participating in Hacktoberfest. I’ve already been contributing more to the open source community over the last few years. Particularly for the Rust programming language. I set out to use this event as an excuse to help the Rust community out.

Positives:

The feeling of a community effort to contribute to Open Source

A typical open source PR

Contributions to open source can happen at any time. I think it’s a nice idea to have a time where people who use open source software are encouraged to contribute meaningfully. Especially for anyone who may feel a bit of intimidation to do so. I’m sure a T-shirt and some stickers can feel very validating to someone who was just using open source before, and now they have some evidence that they contributed too.

Discovering smaller projects

Bob Ross had amazing commit messages

This was a great opportunity to explore Github for projects that don’t require a lot of experience or commitment in order to help a maintainer out. The repos I offered code to were all small Rust projects that were doing some practical things. It was fun, and I put some effort into what I wrote in addition to having good interactions with the maintainers.

Negatives:

This year’s Hot-fix to the rules and the lack of follow-up communication

Talk to your community

I feel like the communication around the event is not being managed well. Very reactive, and not effective at spreading the word.

A few days after the start of the event, there was a significant change in the rules. I only happened to learn about it through the grapevine, as opposed to from Digital Ocean or through the Hacktoberfest front page or profile pages. Rules feel like they are inconsistently applied, and it feels unwelcoming.

Where this change affected me is their change of minimum review times, and probably the validity of my contributions. But I’m confused about what is supposed to apply to me since I had created all my PRs in the first 3 days.

What makes this more confusing is the conflicting information within the Hacktoberfest FAQ.

For example, this is likely some old information about the review time of PRs. States 7 days.

From FAQ. Review time of 7 days

Later in the same FAQ it says it’s been updated to 14 days. But only those before October 3, 2020 @ 12:00 UTC

Also from FAQ. Review time of 14 days

All 4 PRs were made before this rule change was stated to go into effect (Oct 3 @ 12:00 UTC). The latest being Oct 3 @ 6:26 UTC.

4 PRs. All earlier than Oct 3 @ 12:00 UTC

Changes to the rules could have been communicated somewhere on the profile page in the legend, but were not. Or through email, like the message I received when I made my first PR.

I just noticed all my maturation times jump from about a week to about 2 weeks. 3 of these were PRs that were already merged too.

And I’m more annoyed and slightly unmotivated to continue in the event.

(But tbh, rather than feel angry, I’m psyched that I got a few PRs merged so quickly!)

Missed opportunity to communicate that this duration has been updated from 7 to 14 days.

Feeling like corporate-sponsored projects are taking advantage of free labor

I think that the core intention of Hacktoberfest is well meaning. But I'm seeing some non-trivial work in big projects labeled for Hacktoberfest.

I’m not personally driven by the swag of a free T-shirt. I just want to help the Rust community and level up my skills while I'm at it. But I feel weird sitting near loud, negative opinions that I partially agree with. (Although my feelings are a lot more charitable and less intense)

Before the silent rule changes, the premise of projects that must opt-out really placed a huge burden on the maintainers of small projects.

A typical maintainer during Hacktoberfest

But I’m mostly in my raw feelings, because I feel like the focused energy of the open source community is being taken advantage of by bigger corporate entities for a T-shirt. They are all on the bandwagon to use free energy from eager and inexperienced developers to contribute to their big name-brand projects even though they have their own funding for development.

But... will I participate next year?

Hard to say at this moment, but what incentive do I have? I can contribute any time to small projects, and I most likely will. I have to thank Digital Ocean for helping me realize that.

The spirit to drive help to small projects is one of my favorite parts of open source, and I think that Hacktoberfest probably started out with that ethos before their audience made it about T-shirts and stickers.

I don’t like the gamified feeling, and maybe that makes this event not for me. The new rule changes will very much change the feel of participation. Probably for the better, since it is opt-in.

As I continue to dig into the changes, it appears that there was a tweet made prior to the changes, but from my perspective, if you didn’t use Twitter and follow @hacktoberfest, you’d continue to be lost to the change.

This message appears on their site, but I only got to it through the PR. I don't know how else I would have found it because it isn't on the front page or anywhere I'd easily find.

On one account, I've been told that an email was sent out to highlight rule changes. Pretty late after the fact, but I guess better than never. I cannot confirm this myself, since at the time of this writing, I haven't received any email regarding rule changes.

Depending on how the changes affect the mood and the outcome of this year's event, I could probably warm up to the idea of 2021. Either way, I'll still be opening PRs against something.

I'll give credit where it is due to the Hacktoberfest team for responding to abusive participants. BUT I don’t appreciate rules being changed without care to communicate them effectively. I expect better communication from a worldwide event. Please learn from this!

Anyway, if you've made it this far, don't wait for October to contribute to open source!

DEV Community: T.J. Telan

How to Build a Custom Integration Test Harness in Rust

Let’s talk about integration testing in Rust

How does cargo test work by default?

What do I need from integration testing?

Where do we get started with a custom test harness?

Add Setup and teardown steps

Collect all integration tests

Building the test collector

Adding new tests to plugin registry

Running tests from registry

Tips for extending the example

Conclusion

9 Insights I Uncovered While Building a Writing Habit

Building a writing habit

How did it go?

1. The hardest part is getting started

2. Editing is easier than writing

3. Don’t be a perfectionist

4. Writing consistently doesn’t always yield consistent quality

5. Be really specific about who you are talking to

6. Writing titles is hard (because SEO is hard)

7. Keep a backlog of ideas

8. Building structure and motivation for writing is moving target

9. Don’t be afraid to be seen trying

Conclusion

Questions to Find the Right Continuous Integration System for You

What are the goals of Continuous Integration (CI)?

How does CI differ from shell scripting?

What type of CI systems that exist today?

Stateful (WebUI) pipeline config

Pros

Cons

Stateless (File-based) pipeline config

Pros

Cons

My opinion

What are the CI Hosting Options?

Self-hosting

Pros

Cons

System as a Service (Hosted)

Pros

Cons

Issues to Consider when Switching between CI platforms

Threat model

Contributor model

Maintenance

Cost of Build minutes

Responsiveness needs

Technologies used

Delivery/Deployment model

Conclusion

Re: Reading More Technical Content Can Help You Improve Your Content Creation Skills

Why do I like a piece of technical content?

Through social media

Through search

Who is their intended audience?

What does this (company/individual) do really well?

What does this content help me do?

How would doing (or implementing) “X” improve my content?

What can they improve upon to make this content even more effective?

Conclusion

Quick Primer to Practical Reproducible Builds: Reference Environments

Overview

What’s the benefit?

Your reference environment can be used for

Start your reference environment in steps

First level: Automate what is easy and Document what is hard

Second level: Shell scripts

Third level: Configuration management

Next level: Portable Reference Environments via Virtual Machines & Containers

Platform specific caveats

Conclusion

Rust in 2021

What I want for the Rust community in 2021

I want to see Rust shed some of its reputation for being hard to learn

More blog posts from developers and management using Rust at work

More Rust in DevOps tooling

Better Rust Interoperability between languages

How does `cargo test` work by default?