DEV Community: Henry Quinn

How A Free Web3 Project Is Helping Me Build a Locally Focused DAO On Solana

Henry Quinn — Thu, 02 Dec 2021 13:47:41 +0000

Back in 2017, I (and I imagine a LOT of other people on DEV.to) got really into cryptocurrencies. I made some beer money off Ethereum, followed some projects, and really dug into the kind of experiences blockchain technology was able to power.

Everyone I talked to seemed understandably smitten with the concept of decentralized finance, but I was trying to cut through the noise of all the day trading in an attempt to find where actual bedrock innovation was being made.

I sifted through a lot of altcoins and vaporware to find projects like Golem (a way for folks to rent spare computing cycles to artists who need to render their work), IOTA (a network built to handle IoT transactions), and Request Network (a suite of financial tools for crypto payments). These were teams that bucked the trend of trying to make a boatload of money and instead were actually trying to change the way that we interact with the Internet. Something about these teams of people in the middle of one of the craziest bull runs of history saying "screw the quick money, what we're working on is IMPORTANT" really stuck with me.

Then the bubble burst, the hype died down, and I yanked out most of my (meager) investments. I kept a loose eye on the space just to keep up to speed over the years but was hesitant to get involved again.

But somewhere around the beginning of this year, I started looking into decentralization technology that wasn't necessarily on a blockchain, as I didn't love the ecological impact of networks like Ethereum. That led to me spending time digging into IPFS and getting closer with Jake from my local tech Slack who's really into Solana - a blockchain that's fast, cheap, and not burning down the rainforests.

Once I really understood what Solana could do, it felt like a switch got flipped in me.

That's about when I caught a tweet from Nader Dabit about Developer DAO, saw that I had enough ETH left in an old wallet to cover the transaction fee, and just yolo'd it. I bought my first NFT and simultaneously joined my first decentralized autonomous organization (DAO).

The first few days of being part of a DAO filled with people just like me were wild. Everyone in the Discord was trying to learn everything they could from each other about web3 technologies, work on their own separate projects, and then dump all that knowledge and energy back into the group. The value wasn't in a cryptocurrency attached to the project, it was in the community itself. That was like a breath of fresh air.

I love helping run my local tech group but there's a certain freedom in how a DAO allows everyone to make decisions and collectively help shape the future of the thing. Every little proposed change doesn't have to route through a small team of admins. People can contribute development work, documentation, insights, or anything... whenever they have the time. And if something in the DAO needs to change everyone can discuss their viewpoints and vote on a proposal. It turns an online community into a sort of living organism rather than a top-down hierarchy where an organizing team gets to call the shots.

Ultimately, my point is that helping the group comes back to helping yourself.

I distinctly remember sitting back and thinking of my crypto journey around 2017 when I was looking through projects trying to find innovative web experiences powered by blockchain technologies that just didn't make sense using web2 technologies at the time. In 2021, amidst all of the NFT sales going on, I really and truly feel that this new community-owned organization model is the killer app.

Jake pointed me towards some more projects on Solana (surprise!) that shared that same sense of community. I ended up finding groups that welcomed me with open arms and immediately made me feel like one of their own:

Nifty Nanas: This one is arguably where I spend most of my time on Discord. It's a community of wonderfully positive people who want to help support other projects in the Solana ecosystem. They do a lot of giveaways, game nights, and Twitter Spaces (with other web3 projects) and it feels like a real group of friends. Plus the NFT art is rad - just look at my profile picture.
WGMI NFT Collector's Club: A rad group of people doing a series of community art projects where holders of their WGMI token get airdrops throughout the year of NFTs that are part of interactive experiences. On day one of me being involved, I got to hop on a call and help brainstorm ideas for a future airdrop.
Tech Folk: A team who's working on an NFT game based around working together to build startups, upskill "employees", and vying for investors. They've been nothing but helpful in answering peoples' questions, including my own, about web3 development.
Hoodiez NFT: Just a bunch of people hanging out and using NFTs to raise money for both homeless and mental health-focused charities. This is actually the first project where I got to be involved in the initial mint and it's been fun watching the community grow.

About two weeks ago, I remembered one of Jake's tweets from earlier in the month:

I've been wanting to find a way to get more involved in the crypto community and this seemed like the perfect opportunity. I have a few years of running a locally-focused tech community under my belt, and to be quite honest - the members there were getting fed up with the few of us talking about blockchain stuff. Jake had been joking for a while about "wen [local tech group] DAO," and I figured it was time to make it happen.

So after letting him know I wanted to make it real, we (and our friend Rory) jumped feet first into building it.

We fleshed out a Discord server, set up a Twitter account, and built a rad website. We were in the process of talking behind the scenes about what direction we wanted to nudge this new community in before we started inviting people in when I discovered buildspace.

Up to that point, I had been learning bits and pieces of existing projects I could pull together from the Solana ecosystem. But having buildspace provide a built-in community of developers working on similar project prompts allowed me to get my hands dirty with the real thing.

Bear in mind, we're still going to use tools like Grape (for account/wallet verification on our Discord) and Squads (to handle on-chain voting). But I was low-key dreading having to roll my sleeves up and get dirty with the code to learn how any of it worked under the hood.

If you've ever tried building with a new technology, you know the documentation isn't the best if it's even there at all.

That's the vibe I've gotten from most of my forays into dApp development over the years: things are going to be messy and you're going to have to solve your own problems. Not a big deal. That's actually a great way to learn and make all those new concepts stick, but it's not the most attractive first experience with a tool.

The "Build a Web3 App on Solana with React and Rust" project really helped me understand how the websites I've been using actually interacted with both my wallet and the Solana blockchain. I was able to get a dApp for collecting rock climbing gifs that uses Phantom for authentication deployed to Solana's devnet within 24 hours of starting the project.

I know the community management piece required to get a group like ct3_dao off the ground, but I'm helping build a community to help people learn web3 development partly because I want to learn how to code for web3 projects. And buildspace really comes in clutch there.

BUT Y'ALL. THEN THEY DID IT AGAIN.

Right when we were starting to talk about how we wanted to handle minting (free) tokens to let people officially be part of the community (and power the on-chain voting with Squads), I realized I was less than a week out from another buildspace project called "Ship your own custom NFT collection on Solana w/ Metaplex in a weekend".

Through my discussions with folks in Nifty Nanas and WGMI, I knew that Metaplex and Candy Machine were the right move for what we needed (especially considering we wanted to do as much of this on Solana as possible). This buildspace project truly could not have come at a better time.

I like presently have a need to learn how to do use these tools so that I could help build a project that's going to help people in my local community learn how to build for the new web. Buildspace was ready and waiting with content to walk me through a tutorial on how to do that exact thing.

I've seen people talking on Twitter about how they were able to apply what they learned in various buildspace courses to projects they ended up co-founding, but that's quickly becoming my reality. Imagine my surprise when they announced this week that they have another two projects coming up titled "Build your own DAO with just JavaScript in a weekend" and "Ship your own DeFi dApp on Avalanche in a weekend".

Again, we're going to be utilizing Squads to power our DAO but I've never been burnt by learning what's going on under the hood of the tools I use, especially for adding custom additions later. And that Avalanche project is going to be helpful when I carve out time to work on a web3 iteration of my Tech Enthusiast Scholarship.

Ultimately, we were able to soft launch the website and finalize the Discord this past Sunday night. While writing this article on Wednesday, we're already up to 40 local members talking about helpful tools they find online, different projects they want to hack on, and figuring out how we can collaborate with other organizations to help more Connecticut developers make the jump to web3.

We've already made connections with local Makerspaces in Hartford & New London, set up a time to record a podcast episode with CTStartup, and are tentatively planning our first IRL meetup at District in New Haven.

Jake and I had joked for a while about making a DAO to experiment with what a locally-focused community-owned organization might evolve into. But it took a handful of things lining up just right to make it happen. Buildspace having content ready to teach me (and so many others) about Solana development with Anchor, NFT creation and minting with Metaplex, and about the general underpinnings of a DAO structure... all over the course of three weekends (albeit spread out a bit) really helps give me the knowledge and confidence I need to get this project over the finish line.

I definitely could have ground through learning all of this on my own, but dang if buildspace isn't the exact antithesis to my earlier point about documentation being rocky at best for new technologies. They really pushed me in the right direction to help turn a hopeful joke into a real-life community that's already helping people learn.

Me, Jake, Rory, and the rest of the ct3 community are just getting started with ct3_dao, and I can't wait to fill you all in with more technical posts about how we end up implementing everything we want to build.

Until then, stay frosty.

HenryGivesCoffee: Micro-Scholarships for Rad. Web. Stuff.

Henry Quinn — Mon, 05 Apr 2021 17:55:13 +0000

This is a blog post from the launch of my new side project Henry Gives Coffee.

The short version is that I love this new boom we're seeing of folks building personal "non-corporate-y" corners of the web for themselves and I want to encourage fledgling developers to keep building!

Are you working on something rad? Let me throw you some coffee money!

This month's theme is "Web Projects for Local Communities." So if you (or anyone you know) is working on anything even vaguely under that umbrella, head on over to the website to tell me about it! Might just be a free coffee in it for you!

The web is astoundingly more interesting when people are building rad shit.

I was born in 1992, had access to a computer since the age of four, and experienced firsthand the emergence of web 2.0.

The internet seemed like a modern miracle. Plenty of folks have written about how the "early web" was like the Wild West where anything can happen. A free exchange of ideas, information, and art.

In all seriousness, though, that's what it was.

When I was seven years old, from the back of our living room, I was able to apply online for an audition for (and eventually act in) a traveling musical that would go on to have a few mentions in the New York Times.

When I was fourteen, after moving every couple of years for most of my childhood, I was able to use social networks like Facebook to get back in touch with every best friend I've had since pre-school. And let me tell you, connecting those dots was a REALLY fun road trip.

When I was twenty-one, I started learning modern web development in earnest. I wanted to start contributing towards building the amazing internet that I'd always taken for granted. Between free time at work and attending a local web dev bootcamp, I was able to build all kinds of fun stuff.

And now, at twenty-eight, the web is starting to feel REALLY stale.

The web I grew up with was full of personalized spaces that people built to share something they were passionate about:

Some developers built games for everyone to play.
Enthusiasts created static sites to share information about their hobbies.
Others just wanted to share what was going on in their lives, before Movable Type commoditized the idea of a blog.
And honestly, there was seemingly no end to what folks wanted to share.

It really feels like (and data from Alexa backs this up), the content part of the internet (at least in the US) is mostly a conglomeration of Google, YouTube, Yahoo, Facebook, Reddit, Wikipedia, Netflix, and Instagram.

There's a lot of good content there, but it's from a lot of incredible creators on a small handful of platforms that incentivize (intentionally or not) a homogenization of content to please "the algorithm." It often seems like folks can't really be themselves when they're needing to bend their creative processes to match what garners better metrics through the lens of a machine learning backed recommendation engine.

Counterpoint, though: the recent rise in popularity of static site generators and easy hosting platforms seems to be creating a new boom in people creating, owning, and populating their own personal spaces on the internet!

Blogs are back. Enthusiasts are sharing information about their favorite hobbies. Teachers are releasing courses on their own platforms. Even non-technical folks are getting in on the fun.

The internet I grew up with is showing signs of life and seems to be crawling back to being fun and personal and weird.

It's an incredible thing and I want to use the platform I have, however small, to do my part in encouraging this kind of growth to continue.

We're in a period where all sorts of people are creating all manner of cool things on the web.

Statistically, there have to be some really rad projects that aren't getting the kind of distribution and awareness they deserve.

And on every project I've worked on, I've always come back to thinking "wouldn't it be rad if I could earn some beer money from this?".

That's what Henry Gives Coffee is about.

I want to highlight the incredible work that both newer and lesser-known web developers are doing. Hopefully throwing them some money for coffee (or beer) and signal boosting the cool projects they're working on lets them know that what they're contributing to the internet is worthwhile and motivates them to keep building.

I've done okay for myself working on my little projects mostly in the shadows. Hopefully, whatever little nudges of encouragement I can provide will keep people building who might otherwise get frustrated with web development.

And remember, the internet is fundamentally a way to connect all of us. It should be a lot more fun and a lot less corporate. I'm happy to be able to throw this project together and do my part but my hope is that at least one of you reading this might take away some action items and spread the "GivesCoffee" ethos of encouraging folks to both own and personalize their own corners of the web.

I'm pumped to finally launch this project for real and see all of the incredible things y'all are working on.

Hopefully, it's a wild ride.

Stay frosty.

Website Update - Signal Boost Page

Henry Quinn — Sun, 14 Feb 2021 01:07:36 +0000

I just finished giving HenryNeeds.Coffee its annual-ish refresh. I updated the menu bar, split up a lot of the homepage into separate content-specific pages, but my favorite addition is the new Signal Boost page.

When I was looking around for refresh inspiration, I came across Christine Dodrill's website. It has a similar terminal-ish design as mine but with way better colors for everything - which honestly is what I wanted to address with the refresh.

But in looking through Christine's site, I found the Signal Boost page - designed to put a spotlight on other tech folks looking for work. After a year where a lot of people lost their jobs, I felt it was a better use of my time to set up a signal boost page of my own rather than making some colors look better.

There are so many developers, engineers, ops folks, and all kinds of other tech workers trying to land a new gig right now. I'm lucky enough to still be working, so providing visibility to those who aren't is the least I can do with the small platform I have.

I only got to where I am with the help of my friends and colleagues. This is something small I can do to pay that forward, and I encourage y'all to set up something similar.

In that vein, I wanted to share how I threw this together so that you can, too.

First off, this site's code is available in this GitHub repo, but I'll go into specifics about how this particular feature works.

HenryNeeds.Coffee was built using GatsbyJS, and it all sits on top of a hello-world base.

In Gatsby sites, data has to come from somewhere, and I already had certain plugins (like gatsby-source-filesystem and gatsby-transformer-yaml) installed and my ./gatsby-config.js file configured to ingest yaml so that it can be queried with GraphQL.

I had that part of Gatsby's content mesh set up to turn ./src/resume/resume.yaml into content for my Resume and Talks // Pods pages. I expanded that out to power my Blog page with markdown files, and expanded it again to handle Signal Boost.

First off, though, I needed to make a new page so that https://henryneeds.coffee/signal-boost would resolve to something:

// Whole of ./src/pages/signal-boost.js
import React from "react"
import Layout from "../components/Layout"
import SignalBoostLogin from "../components/SignalBoostLogin"

export default () => (
  <Layout>
    <SignalBoostLogin />
  </Layout>
);

All this file does is import and render a component named SignalBoostLogin. That component handles things like important the menu bar, doing some date math for the "Current login" header. But its main job is querying data provided by ./src/signalboost/signalboost.yaml (more on that later) and then iterating over those results to set up individual SignalBoost components.

// Selection from ./src/components/SignalBoostLogin/index.js
<StaticQuery
    query={graphql`
      query signalBoostQuery {
        allSignalboostYaml {
          edges {
            node {
              people {
                name
                tech
                github
                twitter
              }
            }
          }
        }
      }
    `}
    render={SignalBoostPage}
  />

This is the GraphQL query that pulls information defined in that /.src/signalboost/signalboost.yaml file then renders the SignalBoostPage component (in the same file) which ingests the GraphQL results as data.

// Selection from ./src/components/SignalBoostLogin/index.js
{data.allSignalboostYaml.edges[0].node.people.map((person) => (
  <SignalBoost
    name={person.name}
    tech={person.tech}
    github={person.github}
    twitter={person.twitter}
  />
))}

Like I said earlier, this page is just meant to grab the data provided by the yaml file, iterate over it, and generate individual SignalBoost components for each entry via that .map() method. The whole name={person.name} bit passes all the individual data points from the GraphQL results as props that can be picked up and used by the child component (SignalBoost).

// Whole of ./src/components/SignalBoost/index.js
import React from "react"

export default (props) => (
  <div>
    <h3>{ props.name }</h3>
    <p>{ props.tech }</p>
    <p><a href={ props.github } target="_blank" rel="noopener noreferrer">[ GitHub ]</a>&nbsp;<a href={ props.twitter } target="_blank" rel="noopener noreferrer">[ Twitter ]</a></p>
    <hr />
  </div>
)

And this (finally) is the template that takes those props, throws the values into the HTML, and renders out individual divs of name/tech/links on the final Signal Boost page.

So to recap:

The actual data gets updated in the ./src/signalboost/signalboost.yaml file.
The page served by the https://henryneeds.coffee/signal-boost URL calls the ./src/components/SignalBoostLogin/ component.
That component queries the data provided by the ./src/signalboost/signalboost.yaml file, iterates over it, and calls multiple ./src/components/SignalBoost/ components.
Each of those components takes the data passed to it as props and renders out HTML for each person being signal boosted.

So yaml like this:

# ./src/signalboost/signalboost.yaml
people:
  - name: "John Doe"
    tech: "bash docker devops gatsby javascript kubernetes linux sql web"
    github: "https://github.com/username"
    twitter: "https://twitter.com/username"
  - name: "John Doe 2"
    tech: "aws python pandas golang"
    github: "https://github.com/username2"
    twitter: "https://twitter.com/username2"

Will render this:

If anyone wants to add themself, all they need to do is follow the instructions here, edit the YAML file, and submit a pull request.

Once I get the notification, check the formatting, and roll the change into my main branch: builds will automatically kick off on Netlify and Fleek to deploy the updated version.

It took me a couple of days' worth of free cycles to figure this all out and get it working the way I like it, but the current version works great!

So far I've had two folks submit PR's and it went off without a hitch. They submitted their PR, I hit the "Merge" button, and the builds kicked off all on their own.

Building the feature was pretty painless given that I already had my site built on Gatsby's engine. However, adding something like this to a different static site generator or build process should be fairly easy once you understand the data flow.

Lots of folks lost their jobs over the past year, and really what's the point of having a voice, site, blog, or whatever if we don't help others climb up behind us on the same ladders?

Just something to chew on.

Stay frosty.

https://henryneeds.coffee (IPFS Version)
Blog (IPFS Version)
LinkedIn
Twitter

IPFS - The Decentralized Web Is Finally Here

Henry Quinn — Fri, 29 Jan 2021 14:56:21 +0000

It's the spring of 2011. I've already been accepted to college. Senioritis is setting in and 3OH!3 is blaring on my headphones as I sit at my desk at 2 AM on a school night...

Homework long since done, I'm staring at the screen of the heavily used 17" Gateway laptop I managed to get off my boss at The Computer Shack in place of actual payment.

I'm trying to test a tool on a Windows XP partition so that I could confidently install it on my high school's network.

I had already hidden a handful of games on a shared drive meant for our teachers' math and science tools.

My friends and I had plenty there to keep us occupied at the end of long lab periods, but we wanted to be able to chat with each other while we fragged each other in Blood Gulch.

And of course, phones weren't allowed.

This tool would let us piggyback on (the since sunsetted) net send command to effectively let us private message each other over the local network.

Being the "nerd" of the group, it was my responsibility to get this set up. The IT guys may not have loved what I was getting up to, but we had an understanding. I could bend the rules a bit as long as I let them know what parts of their security they needed to shore up.

But this tool, though.

Someone wrote an entire chat application written on top of a network command, put it online for free, and I had the ability (and unintended privileges) to do something rad with it and make the tail end of high school more exciting.

I remember explicitly taking a moment to sit back to think about how god damned cool that was.

Now, about 10 years later, I'm living that moment all over again.

Work was done at 5:00 today. My girlfriend, tired from nursing school, is in bed. Dishes are... well, sitting in the sink dirty.

I'm sitting at my computer: late at night, working on a project, and thinking about how lucky I am to be alive and a few years into my development career right now.

Last week, I stumbled onto a Verge article about a new (well, 5 years old) technology that's just starting to pick up steam, called InterPlanetary File System (IPFS).

Folks who came to this post from a Tweet or something probably know what I'm talking about. But for those who don't:

Wikipedia defines IPFS as

...a protocol and peer-to-peer network for storing and sharing data in a distributed file system. IPFS uses content-addressing to uniquely identify each file in a global namespace connecting all computing devices.

IPFS allows users to not only receive but host content, in a similar manner to BitTorrent. As opposed to a centrally located server, IPFS is built around a decentralized system of user-operators who hold a portion of the overall data, creating a resilient system of file storage and sharing. Any user in the network can serve a file by its content address, and other peers in the network can find and request that content from any node who has it using a distributed hash table (DHT).

Long story short, the future of the internet is here.

A group of people worked for years to actually make the decentralized internet that Pied Piper was trying to make on the later seasons of Silicon Valley.

They built an entirely new protocol for accessing the web. It's meant to work alongside (not replace) HTTPS and operates similarly to BitTorrent in that data (for websites, videos, games, apps, anything) is shared amongst peers and not fetched from central servers owned by Amazon or some other Evil Corp™.

The ramifications here are HUGE.

1. Have you ever had a resource on the web disappear on you?

That's a dumb question because so have I.

There's this mashup called "Ketchup V1" from an artist named The Deaf DJ that I'd love to be able to find again.

I'm sure you all have your own white whales.

Content that we care about on Web 2.0 (the HTTPS based one we're on now) only sticks around if the people creating and hosting that content want to keep it available.

If some publisher or individual decides they want to take a site down or stop paying for a monthly web server bill, then whatever information they were hosting effectively disappears.

The internet is supposed to be the culmination of all human knowledge, but small and easy to host files (like sheet music or a set of woodworking instructions) could disappear because someone needs to reign in their budget.

IPFS allows anyone to go "Hey, that blog post on how to code this feature was really helpful to me, I'd like to re-host it so that other people will always be able to find it later."

The Brave browser, mentioned in that Verge article, implemented IPFS features that let me do that with two mouse clicks.

That's a beautiful thing.

2. Speaking of resources disappearing, did you know that for a few years Turkey blocked their citizens from accessing Wikipedia?

Again, the culmination of all human knowledge, but sequestered from citizens.

Kind of fucked up, right? Especially when you know the importance of tools like social media sites during times like the Arab Spring. Being able to block entire portions of the web, like China's Great Firewall, is a dangerous precedent.

Turkey was able to block Wikipedia because web addresses are currently based on file locations.

You tell your browser to go all the way to the group of servers that the Wikimedia Foundation set up to host Wikipedia and then, once there, pull up the page on water filtration.

Since Turkey could see where those web requests were pointing (the addresses assigned to Wikipedia servers), they were able to stop them from completing.

Since everything on IPFS is content-addressed (via a hash), whenever you request a page, your browser pulls it from whoever happens to be closest to you and is also hosting that file.

With this new model of web browsing, we're not dependent on corporations renting servers from even bigger corporations to keep the information we care about, and they currently control, on the internet.

Instead of relying on publishers like the New York Times keeping articles hosted on their servers, or platforms like Stack Exchange seeing the value in keeping answers to years old questions online, we can just click a button to rehost web content so that it's always accessible on a massive P2P network.

No more going all the way to specific servers that governments, ISPs, or rogue actors may be able to cut off access to in the future.

This gives the people the ability to decide what's worth keeping around in perpetuity. Theoretically, the good stuff will be pinned by enough users that it'll be around forever, while the bad stuff will fall to the wayside.

3. It harkens back to the markedly non-corporate early internet so many of us fell in love with.

I, for one, am tired of the internet being mostly a collection of the same seven sites over and over again.

So many of our favorite old websites and interesting things we bookmarked just don't exist anymore.

All of that effort and content and knowledge is just lost forever, only to be replaced by "community of communities" sites like Facebook and Reddit.

I miss the days where we'd go to AddictingGames to play some games, check out some tech news on whichever blog we liked, get involved in separate niche community forums, and generally exist without having to move all our data through massive corporations.

This move to a decentralized web has been encouraging early builders to make fun little web pages, more niche community apps, and just experimenting with the new format.

It's like the wild west on the web again, and while a lot of the tooling and architecture still needs to be figured out, the possibilities are endless.

The new web is weird and that's a good thing.

That brings me back to my earlier point:

I'm sitting at my computer: late at night, working on a project, and thinking about how lucky I am to be alive and a few years into my development career right now.

This kind of decentralized web that lives on all of our devices and frees us from our dependencies on corporations is something me and my computer lab buddies have wanted to exist for a long time.

Over the years since college, I've headed up tech communities, led nationwide teams of developers, and traveled around the country giving and listening to talks.

All that and I've honestly never been more excited about a new $web_thing crossing my desk.

IPFS is the real deal. From nerd dreams of old to HBO jokes to your ears (well, eyes).

It's finally here, it's being given away for free, and I understand enough modern web development to help build it.

That's something 18-year-old me would be absolutely hype about. I just hope I can make him proud.

I'm working on a project built on IPFS with a buddy of mine from GitLab.

Without giving too much away: it's built with GatsbyJS, hosted on IPFS with Fleek, and should help folks adapt to the early days of this new web protocol.

I should another blog post with some more information to share soon.

Stay frosty.

https://henryneeds.coffee (IPFS Version)
Blog (IPFS Version)
LinkedIn
Twitter

Hacktoberfest Markdown Editor Challenge: Day -2 (Prep Work)

Henry Quinn — Wed, 30 Sep 2020 13:32:19 +0000

Real quick, as an aside:

As an aside - in the spirit of Hacktoberfest, y'all should all check out All Things Open - A polyglot technology conference focusing on the tools, processes, and people making open source possible.

It's a free virtual conference taking place on October 19th and 20th, with tracks covering topics like DevOps, community leadership, inclusion and diversity, and various workshops.

It's going to be a great time full of great talks. I'll be giving a full-length talk on tech debt and an ignite on digital transformations in government myself, but I'm very excited to hear from folks like Liz Fong-Jones, John Papa, and Remy DeCausemaker.

If you're reading this, I'm assuming you saw my last post on how I want to bump up the Hacktoberfest challenge a bit above just making four pull requests.

I want to build my own open source cross-platform cloud-synced desktop markdown editor in just 31 days.

Henry, why do you keep doing these things to yourself?

Great question. I'm unsure of the answer, though. I find that when I get stuck on a problem like this for so long (at this point a couple of years of trying to find a markdown editor that fits all my needs) that I just need to buckle down and solve it. It's mostly just a fun bonus that it gives me some material to blog about on DEV and to stream on my Twitch channel (Tuesday nights at 6!).

Much in the same way that novelists prepare a bit for National Novel Writing Month (NaNoWriMo) by figuring out their characters, sketching out the plot beats of their story, and spending some time thinking about the hell they're going to put themselves through in November... I feel that it's appropriate to prep for this markdown editor a bit.

You wouldn't jump into a half marathon without doing a few half marathons first, right? Well, I might have in 2018, but that's a story for a different time.

Let's get right into it: this is a BIG project to take on in just a month. For a refresher, here are the big bullet points I want to hit by the end of Halloween:

Being cross-platform (Linux/Mac/Windows and eventually Android/iOS)
Ability to cloud sync data between those platforms
Having one editor pane where markdown syntax is rendered on the spot (like Bear and Typora)

You idiot, you didn't even put "build a markdown editor" on your list.

Hey, thanks! That's the first thing I wanted to talk about.

Much like writing a book, or running a marathon, building this app is going to be a slog - even if I'm just aiming for an MVP that I can keep iterating on. I'll be proud as hell if I can get through those main three items without getting too far into the other 20+ feature ideas I want to bake into this thing.

But the truth is that the actual markdown editor part of it is pretty easy. Part of my prep work for this was Googling around for some "electron markdown editor tutorial" tutorials. I wanted to get my feet wet both with how Electron apps are put together and with what a markdown editor might look like in JavaScript.

After trying a few, I found that one written by Tzahi Vidas was the simplest that both showed me how to build a simple Electron app and how to parse markdown with JavaScript. I highly recommend y'all give it a shot if you're at all interested in what I'm working on. It's a solid primer.

Tzahi Vidas - Here's how I created a markdown app with Electron and React

I did, however, find that I had to use a different command to run Electron apps on my MacBook Pro than the one provided in the above tutorial, though. In package.json, I had to change the run script from something like electron . to electron-builder build --mac -c.extraMetadata.main=build/main.js --publish never to get the app to actually launch. Odd, and took me a bit to figure out (source on the fix - thanks John Dyer!), but it was a solvable problem.

In a bit of backward thinking, I then went on to checking out the Electron docs to see if they had any getting started docs. Turns out they have all kinds of cool nuggets in there, but some of them are a bit buried in an interesting hierarchy of links and pages. Within there, I found two really helpful things:

The Electron "Simple Samples" GitHub repo has a few sample projects already built out that interact with your computer's resource monitor, your app tray, and some other bits of their API. You can just run npm install and npm start to pull one of the projects up on your local machine and dig around in the code to see how it all fits together. They even give you a set of challenges per sample project to try and add functionality.

* Electron Simple Samples Repo

The second helpful thing I found probably would have been better off if it were the first, even before Tzahi's tutorial - the "Electron API Demos" repo. When you npm install && npm start this bad boy will pull up a window telling you all about the different parts of the Electron API you can use to interact with a user's desktop, has buttons to show on your desktop what each one does, and has code snippets to show you how to use them.

* Electron API Demos Repo

Between those, and digging through the Electron docs a little more, I got most of what I needed to get ready for this challenge. I have a cursory understanding of how Electron apps work, how to parse markdown with JavaScript, and feel mostly ready for October. At least as prepared for it as writers are for NaNoWriMo or runners are for a marathon. I know the basics of what I'm taking on, but the event itself is going to bring plenty of its own challenges.

There are still a handful of things to figure out as I get started in October.

Like, am I going to use an existing markdown library, or am I going to make my own parser with slightly altered markdown syntax rules?

How do I handle the cloud syncing: through something like PouchDB or by treating the whole thing as a progressive web app and use service workers to keep local offline changes synced with a SQL database somewhere?

On that last point, a buddy gave me some words of wisdom today:

"I would start with the simplest possible solution, probably some form of long-polling, evaluate the UX once the app was a further bit along, and reassess then." - Ben J, a SWE and former tech co-founder

And I mean, smart. Knowing myself, it would be all too easy for me to forget that I'm just aiming for an MVP within like three days. I'll always have time to add features later.

Past that, there are all kinds of things I'll have to figure out as problems pop up all through November, but I'm glad I did the prep work I did to get myself ready to tackle this whole thing with a little prior knowledge.

Tomorrow is day -1. My last "day off" before the development work starts. And I also have a new conference talk (titled The Tech Debt of Monopoly House Rules - it's gonna be a fun time) due in a few weeks for All Things Open. It's going to be a busy month for sure, but I'm excited to get some work done. I'm going to cook something fun tomorrow, relax a bit, and get back to y'all with an update on October 1st.

Until then, stay frosty.

https://henryneeds.coffee
Blog
LinkedIn
Twitter

How would you handle cloud syncing data for an Electron app?

Henry Quinn — Tue, 29 Sep 2020 16:10:55 +0000

So I'm going to be building an open source markdown-editor-of-my-dreams for Hacktoberfest and one of the things I want it to do is handle cloud syncing note data across multiple platforms.

App is going to be JavaScript, Node, Electron.

I'm not sure how to handle the native cloud syncing, though.

My gut reaction was to just toss it all in Firebase.

I think a better approach would be to treat it like a Progressive Web App and use service workers to keep track of local offline changes and update a SQL database whenever a user reconnects to the internet.

How would you handle this?

Hacktoberfest 2020: Building a Better Markdown Editor

Henry Quinn — Sun, 27 Sep 2020 18:24:51 +0000

What's up, y'all? Long time, no... blog?

Hacktoberbest is almost upon us and this year I want to take things a little farther than just submitting a few pull requests. The event is meant to help people get more into open source development, and in that vein I wnat to treat October the same way authors treat National Novel Writing Month (NaNoWriMo).

I want to start and finish a useful project within those 31 days.

I owe a lot of my career to folks putting their open source projects, packages, and products on the internet for everyone to use and I want to pay part of that back to the community I've gained so much from.

In the past, I had a lot of success on DEV.to when writing my series on Learning DevOps In Public (which itself was inspired by Shawn Wang's post). Making sure I was able to write clearly about what I was learning and applying turned out to be an incrdible way to learn, and it turns out that people vibe with that content! That series had a cumulative 13,500 views!

After having a good cadence in getting posts up for a while, I got caught up in the busy season at work and then the world kind of... blew up. Kind of fell off the grid for a long time and missed out on a lot of moments of good community building and interaction.

Honestly, for a while I was doing my best just to keep my head above water. There were projects I wanted to hack on but between work, cooking, and trying to find new ways to spend time with my friends... I didn't have the energy. I've been wanting to get back into tech writing and project work but needed the right idea to come along and the right motivation to get back to it.

Finally found the next thing I'll learn in public, and just in time for Hacktoberfest:

Building a better markdown editor.

I know I'm one of thousands (likely more) developers to take a whack at making the "perfect markdown editor", but hear me out.

My buddy, Alex Trost (curator of the Frontend Horse newsletter - which you should all check out), and I have been trying out different markdown editors over the last year or so and while the 85% of their features have a solid overlap, it's often the other 15% that we love about each individual editor.

Bear (what I currently use on my work MBP) has a fantastic layout and organizational system, but doesn't support anything other than macOS and iOS.
Typora (what this post was written with) has solid cross platform support, but doesn't have any native cloud syncing functionality.
Other editors have WYSIWYG bars (not really markdown), some are web based (not ideal for me), and still others cost money when they feel feature incomplete or have stale codebases.

After doing a survey of a ton of different options, I landed on my dream editor having three main features:

Being cross platform (Linux/Mac/Windows and eventually Android/iOS)
Ability to cloud sync data between those platforms
Having one editor pane where markdown syntax is rendered on the spot (like Bear and Typora)

It seems odd to me that I didn't come across a mainstream markdown editor that covers all three of those points. Maybe I'm getting in over my head with this project, but I feel like this is a solvable proble, you know?

I've been wanting to dig into Electron for years now, and I'm sure plenty of other web application developers feel a similar trepidation of making the move over to desktop applications (even if it is the same technology in the background). There are a number of tools I've built with JavaScript for my job, and being able to wrap a GUI around them quickly would make it easier for me to share them with folks who feel less at home on a command line.

After running through a couple of Electron tutorials (which I'll write about early on in October), I found that it's a pretty simple technology to use if you already have some familiarity with Node projects. I hope that by writing about my development process I'll be able to help some of y'all make the jump from web to the desktop.

A good markdown editor obviously means more than those previous three bullet points to help boost productivity, though. I whittled my wishlist down to a "top 20" list of other features (in order of importance to me):

Local storage in something like SQLite
Left side bar for list of notes (title, first couple lines preview)
Auto save
Add todo's/task list with Bear's - syntax
Code blocks (MarkText uses GFM code fence, syntax highlighting - PrismJS?, line numbers)
Syntax support for popular programming languages
Word count (word/characters/paragraph/read time)
In-line styles (like strong, strikethrough, underline, comment, etc)
Table of contents generated by headers
Show creation/edit date and last editing device
Full in-line image support
Table blocks (MarkText uses GFM table block)
Shortcut keys for styles
Focus mode - new note in Bear
Light/dark modes
Project bundle support similar to FastAuthor (https://github.com/ExamProCo/fast-author#The-Anatomy-of-a-Project)
Export as different file types (HTML/PDF/MD)
Organize notes with hashtags?
Ability to cross-link and reference other notes
Encrypt individual notes and lock the app

It's an aggressive project to tackle in just a month, but I don't see myself getting too deep into my backlog of wishlist items. Figured that having a bigger project to tackle during the month of October would help keep me motivated and make it feel like I actually earn my t-shirt and sticker pack this year - and leave me plenty to do while I try to flesh this app out through the end of the year.

Plus, there's the added benefit of me being able to use a tool I've been wanting for a while and getting full creative control over it!

I plan on working throughout the month to get at least an MVP put together and want to get a post up every few days on what I've been doing. I learned in the last round of learning in public that posting daily was too lofty a goal.

But I'm hoping to use this opportunity to really dig into using Electron to build desktop apps, get back into writing, and hopefully become a better developer while taking y'all on this journey with me!

Here's to tackling something big in 2020, and I'll see y'all on October 1st!

Until then, stay frosty.

https://henryneeds.coffee
Blog
LinkedIn
Twitter

Organizing Local Tech Scenes, A Podcast Episode

Henry Quinn — Wed, 21 Aug 2019 16:24:17 +0000

Hey everyone! I haven't forgotten about all of you. Just had a wild week and a half at work that we're just now getting on top of.

There are still some posts in the pipeline about building, configuring, and running applications on my Raspberry Pi Kubernetes cluster!

Until then, I thought it would be worth sharing an episode of a podcast I was on this past weekend.

Between Two Rocks Podcast: Episode 22 - NewHaven.io

In my free time, I help run a local tech group called NewHaven.io. Several years ago, this group was created to pull all of the disparate Meetup groups in town (different languages, stacks, technologies) under one banner. I've been involved as a board member for about a year now and it's been a wild ride.

We were invited onto this podcast by another of our board members and we discussed how we want to grow out our local scene and focus more on inclusivity and diversity.

Hopefully, this might provide some nugget of wisdom to any of you out there who might be wanting to get more involved in your local developer community. Whether you want to help organize more events or stand up a Meetup group of your own, rest assured that there are developers in your area who will greatly appreciate the effort you put in to help grow your scene.

For all of you already helping run similar groups, what are you doing to provide more value for your members? We're always trying to find ways to do more with less help organizing, so I'm curious about anything and everything you're doing regarding throwing events, growing a sense of community, and fostering a support group.

Again, Raspberry Pi cluster articles inbound! Until then, stay frosty.

https://henryneeds.coffee
Blog
LinkedIn
Twitter

What kind of DevOps content do you want?

Henry Quinn — Fri, 02 Aug 2019 15:39:43 +0000

Hey everyone!

If you've been following along with my Learning DevOps In Public series, you know I have some content in the pipeline. Mostly about building out a Raspberry Pi K8s/K3s cluster and about how to migrate your Docker Compose projects to Kubernetes manifests to Helm charts.

But, since there seem to be a lot of people who are interested in starting to work with containers, I'm curious what you all want to read about!

I've had to teach myself everything from writing my first bits of PHP to getting applications ready for a production OpenShift environment, and I really don't want any of you to have to experience that pain alone.

If there's anything you're interested in learning about or need some kind of clarification on, throw it in the comments. We can start a discussion and I'm sure people can pop in to answer questions in the moment, but it'll also give me an idea of the kind of content that will help everyone here become stronger developers!

The Promise of Docker Containers

Henry Quinn — Thu, 01 Aug 2019 10:02:35 +0000

Hey folks! My last post, titled How To Incorporate Containers Into Your Daily Duties, was all about how you can use containers to improve your work as a developer.

The problem with that is that it necessitates y'all knowing how to set up, configure, and run Docker containers.

I shouldn't have assumed that everyone came to that post with the prerequisite knowledge, so here's an article I wrote a little over a year and a half ago. It covers two things:

What benefits containers provide you, as a developer.
A demo/walkthrough of building an image based on a CentOS LAMP stack from scratch and spinning up a container to serve a web application.

I've also updated the additional resources at the end of this one so that they're the current links/documents/installers as of July 31, 2019.

Hope you enjoy!

The Promise of Containers

As a web developer, I’m sure you’ve built a server or two. You probably spun up a Linux box and installed some packages like Apache, MySQL, or PHP. Maybe you pulled some of your code from GitHub, threw a database together, and edited some config files to bend the server to your liking.

(If you haven’t built a server and are interested in learning how, check out my LearnToProgram.TV course titled “Introduction to Server Administration”. - https://www.udemy.com/introduction-to-server-administration/)

But what happens if your server becomes corrupted, your backups have been failing without you noticing, or your site goes viral and traffic starts outpacing what your infrastructure can handle?

If you’re new to server administration, any of those events could seem daunting. If one occurred, you would have a bigger problem than manually rebuilding your server. You also have to hope that your documentation is thorough enough to cover every modification you made to your original system. You do have documentation, right?

Simply put, a disaster of any magnitude would make for a frustrating day or a late night. Disasters do happen and it’s prudent to be prepared for them. Let me introduce you to Docker, which is a container management system.

“But Henry, what in the world are containers and why should I care about them?”

Great question. Containers, in the context of software development, are best defined as segmented and sequestered user-space instances existing on top of a kernel.

“Yeah… can you make that simpler?”

I sure can. The easiest way to explain containers is to compare them to virtual machines (VM's). One method to run three different apps on one server is to run three different VM's, which consist of an operating system, the app, and whatever dependencies that app needs in order to run. A container, in the context of this example, is a bundle of the bare minimum of code and dependencies it takes to run an application. Docker functions as the guest operating system for any number or type of containers. The obvious benefit is that it is much more resource-efficient than running several VM's.

Source: https://images.idgesg.net/images/article/2017/06/virtualmachines-vs-containers-100727624-large.jpg

"Word. This is starting to make sense. Paint me a word picture."

You got it. Let's dive in on how Docker (and containers in general) are going to take your game to another level. The benefits of using Docker are fiftyfold, but for the sake of brevity, I want to key in on the five which I believe are the most important.

1. Speed

The main key to wrap your head around is that Docker allows you to programmatically build out your infrastructure. That means instead of having to manually build another server every time you need a new one, you can just spin one up based on a predefined image of what you need the resulting container to be. It's a little more time-intensive to build that image at the beginning of a new project, but every time you need to stand up a duplicate environment, it takes minutes instead of hours.

2. Ease of Use

In addition to speed, the ability to quickly spin up instances of infrastructure you need at any given moment can be a great boost to your organization. Let's say the app you're building hits the front page of Reddit and your traffic goes out of control. You could also get hacked and have your site knocked offline. Either way, you need to get back online quickly. Your business very well might depend on it. At least nine out of ten times, starting a container is faster than building a fresh server.

3. Security

A basic tenet of containers is that they are walled gardens. An initial benefit is that you can bundle apps with their respective dependencies so that if container A needs PHP5 and container B needs PHP7, that can be handled without having to worry about dependency clashes. The two containers may run side by side, but they won't get in each other's way.

The walled garden approach also means you can define exactly how you want and if you want data flowing in and out of a container. You'll see in the demo that we explicitly tell our container to "EXPOSE 80". That is saying it is okay to open up port 80 in the container to the broader system we're running it on so that Apache can do its job and host content over that port. When you start building more complicated systems with Docker, you can use that kind of security to link certain containers running applications up to other specific containers holding databases or running some specific micro-services.

4. Portability

Since containers (in a simplistic view) are meant to be an app packaged with the bare minimum of resources it needs to run correctly, you can run that container on anything that can run Docker. Whether it is on Red Hat Enterprise Linux, your MacBook Pro, or Windows Server 2016, that container will spin up and function exactly as it should.

An added benefit is that you can develop in an environment that will eventually be your production environment. Your containers will be based on an image, which is a compiled list of instructions on how to build a specific environment. You can use that image to spin a container up on your laptop and build your app in it. When you're ready to launch, you can use that same image to deploy a container on a public server and be completely confident that it'll run the same there as it did locally.

5. Version Controllability

Finally, since Docker lets you build out your infrastructure as code, your infrastructure can be entered into version control system. One more time for the people in the back. YOUR INFRASTRUCTURE CAN NOW BE VERSION CONTROLLED. If you make a change and something breaks, just roll it back. If your data center somehow burns to the ground, you can pull your images and code from your remote repositories. Future you is going to thank present you for making their life that much easier.

Docker Demo

With all that said, let's build our first container. I'm assuming that I've convinced you of Docker's usefulness enough that you've already downloaded and installed it on your system. If you haven't yet, please go do that now.

https://docs.docker.com/install/

Just for laughs, let's say I need a basic site running on a CentOS server for a demo. There are two ways I can go about that: I can spend the twenty minutes it can take to create a new server from scratch, or I can use Docker to build one image that can then instantiate innumerable identical containers to run my site.

1 - Create a Dockerfile

First, make a new project directory and create a file named Dockerfile, which is a list of instructions on how to build a system on top of a base image. Base images can be based around languages like PHP, run-times like Node, or operating systems like Ubuntu. They are generally provided by the maintainers of the source product. That means they're built by the creators of the software specifically to work well with Docker.

We then need instructions on how to build on top of that base. There are commands we can use, like: RUN, ADD, and CMD. They run shell level commands, map files from the host to a container and specify commands to run on boot, respectfully. We take the steps of configuring a server and translate them into tasks that Docker can use to automate the process.

It's similar to how Git works behind the scenes, in that each commit (or instruction) is a set of changes to be made to the last commit, not the entire codebase at that moment.

For the sake of simplicity, we're going to be building a very basic image. our Dockerfile will be as follows:

# Dockerfile
# Basic Setup

FROM centos:7
LABEL maintainer="henry@tld.com"

# Update repos and install httpd
RUN yum -y update && \
    yum -y install httpd && \
    yum clean all

# Expose a port from the container to Docker and run the startup script on launch
EXPOSE 80
CMD ["/usr/sbin/apachectl", "-DFOREGROUND"]

A Dockerfile reads surprisingly like English. All it is saying is that we want to use CENTOS y as a base, install Apache, expose port 80 to map to a different host port, and start-up Apache on boot.

2 - Build Your Image

Once the Dockerfile is saved, we'll run a "build" command in the terminal/ This tells Docker (the utility) to use our Dockerfile (the list of instructions) to build a system.

First, we need to open our terminal and enter the directory that contains our Dockerfile. For me, that directory is ~hquinn/dev/forDevTo/TheFutureIsDocker/DockerDemo/.

We then need to tell the Docker utility to build the image being described by the Dockerfile.

The --rm denotes that we want to automatically delete the intermediate containers that are generated during the build process, the -t hquinn/dockerdemo is tagging our build with a name so we can more easily refer to it later, and the . at the end indicates that we want to run this command in the present working directory.

This command will log out every part of the build process to the console. It may take a few minutes, which is totally normal. Let it sit for a minute; it will tell you when it's finished.

3 - Verify Your Image

Once Docker pulls down the base we chose to use (CentOS 7), installs Apache, and completes all other instructions, we're given a finished image that is named based on the tag in the build command. You can see it (and the CentOS 7 base image) by listing the Docker images on your system.

4 - Spin Up A Container

Since the image is now built, we can use it to spawn a running container. We're going to change directory to the project we want running on this first container and then run a docker run command.

(Here, I'm using a sample project from Dave Moran's LTP course titled "Learn SASS and SCSS" - https://www.udemy.com/learn-sass-and-scss/)

This command is telling Docker to run a container-based on the image that we just build, but it was a few other parameters. The -d is telling Docker to run the container as daemon (in the background), the -p 80 is to map port 80 on the container to a port on the host, so it can be viewable via a browser, the --name Project1 is again a way to name the resource, and the --mount type=bind,source="$(pwd)"/app/,target=/var/www/html/ is telling Docker to map our local ./app/ directory to /var/www/html/ in the container so that Apache can host it for us.

5 - Verify Your Container

Now that the container is running with our code loaded into it, we can go see it in a browser. First, we need to figure out what local port the container's exposed port got bound to.

We can see that the running container named Project1 has mapped our local port 32769 to port 80 on the container (the host port may be different on your machine). If we open a browser and head to localhost:32769, we can see our code running live.

6 - Drop Into Your Container's Shell

That's all there is to it. We could keep moving to different directories that have basic HTML/CSS/JS applications in a subdirectory called ./app/ and keep using that same command to spawn more containers. They'll each spin up, map that code to /var/www/html/, and assign a port on localhost to each container.

We can verify that the container is pulling in our code by running a Docker command that grants us shell access to it. It'll drop us right into the root of that CentOS system as the root user. If we look in the web directory, we can see that it pulled our files in and uses Apache to host them, the same way would on a traditional VPS.

7 - Clean Up

What demo would be complete without some file cleanup? If we keep spawning containers without doing any sort of maintenance, we will eventually hit the limits of our hardware. That's not ideal. Let's shut down and delete the container we spun up, then remove the image it's based on.

Containers are ethereal. Once one is stopped with the docker stop command, we could spin it right back up, and it would be identical to the first time that we started it. To get rid of a stopped container, we'd run a docker rm command. To get rid of the image it was based on, we would run a docker rmi command.

Outro

Docker isn't as insurmountable a technology as it seems. It can be tricky to wrap your head around, but once you do, it's pretty easy to see the benefits that it can provide. Not only does it let you develop locally in the same environment you'll use for production, but it lets you spin your infrastructure up and down to suit your needs: whether it's scaling to meet viral demand or recovering from a disaster incident.

Furthermore, this is just a very small taste of what Docker can do. Your builds can be as simple or as complicated as you need. For instance, I have one build that consists of a group of three containers: one running a Node application, another container running PostgreSQL with data persistence through the help of a data volume, and the third acting as a query cache.

As a developer, Docker is an incredibly powerful tool to have in your arsenal. Honestly, even if it's meant to solve a problem you haven't had yet. You'll eventually find yourself in a situation where Docker can save your bacon, and when you do, you'll be ready for it. The documentation is incredible and it sounds corny, but the only real limit is your imagination. Now, get started building your infrastructure as code.

Additional Resources

Article & Corresponding Code
Docker Desktop Edition Installers
Docker Getting Started Guide
Docker Labs - Katacoda
Dockerfile Documentation

Containers can be a hard subject to initially broach, since, from the outside, it can seem that there is just too much in that realm to learn. However, once you dig in, it's just some light system administration work that's being automated so you only have to put in a decent amount of effort the first time.

Hopefully, this post is enough to help at least one of you start your journey towards container enlightenment!

Also, the parts for my Raspberry Pi K8s bramble are all slowly coming in the mail. Just waiting on my cooling rack tomorrow afternoon. Can't wait to start putting that together and creating some more content for you all.

Until then, stay frosty.

https://henryneeds.coffee
Blog
LinkedIn
Twitter

How To Incorporate Containers Into Your Daily Duties

Henry Quinn — Mon, 29 Jul 2019 13:30:28 +0000

We're going to take a bit of a sidestep this week.

I was planning on converting a Docker-Compose project into a Kubernetes one, but I had a small project pop up this week. That project involved me setting up a container to mimic a production environment so I could test my script without disrupting any of our clients.

Figured a practical example of how a container saved the day would be a better topic than me learning something just to learn something.

Every once in awhile I'm given a task that, once explained to me, seems like it's going to take a lot of time. Like hours of my attention every time this bit of work comes up.

I'll usually ask how often this task comes up in a given quarter. If the combined time for a quarter is more than a day or two, I usually throw the XKCD comic on automation out the window just go for it.

Photo Credit: https://xkcd.com/1319/

It's more worth my time to write and maintain an automation script than it is to have me out of "the fight" for a few hours to deal with something mundane while the rest of my team is tackling a production issue.

This week I was given a task that requires me to hop on a server and programmatically clean out some files. Sometimes it's just looping over a list and using some simple * matching to nuke some stuff. Sometimes it needs some sed magic to make things work.

Once I decided that it was worth it to automate this task, I copied the Confluence outlining steps into a new document in VS Code. I chunked up the tasks based on similar work that needed to be done, played around with some Bash scripting, cried a bit, and eventually came up with a more sophisticated version of the following GitHub repo:

PROJECT CODE: Release The Hounds <-- GitHub Link

Now let's dive into what exactly is going on in there.

The README is mostly just telling you how to start the container and run the scripts in the ./code/ directory:

$ docker pull centos:5.11

$ docker run -v `pwd`/code/:/var/theHounds/ --name DogHouse -it centos:5.11 /bin/bash

$ cd /var/theHounds/

$ ./SetUpEnv.sh

$ ./ReleaseTheHounds.sh

"But Henry, why are you using a container for this?"

Great questions! Presently I have three answers for you:

We, like a lot of companies, run Red Hat Enterprise Linux in production. Spinning up a CentOS (downstream project from RHEL) server allows me to closely approximate the environment I want my script to run in. This gives me a more similar testing environment than say, Windows.
Containers are meant to be ethereal. One of the scripts in ./code/, SetUpEnv.sh creates dirs and files for me to test against. If I need to test again, I can restart the container, and my setup script will create the same env every time.
Containers don't have access to any local storage that you don't explicitly give them access to. That way, if I mess up my script, I won't accidentally delete a local directory I want to keep.

# EXCERPT FROM SETUPENV.SH:
#!/bin/bash
mkdir -p /opt/hoodlums

touch /opt/hoodlums/2019-03-07_Thief.zip
touch /opt/hoodlums/2019-06-12_Ruffian.zip
touch /opt/hoodlums/2019-07-25_Vandal.zip

So, once I start up my container and run this script, I'll always have the same environment to test my script against. And that's where ./code/ReleaseTheHounds.sh comes into play.

This script exists solely to help me get rid of some files. Sometimes that can be as easy as looping over a variable and shredding any files that kind of match.

#!/bin/bash
# Set up vars
HOODLUMTYPES="Thief Ruffian Hooligan Delinquent Vandal"
...
# A HOODLUMS
cd /opt/hoodlums/
for i in $HOODLUMTYPES; do shred -n 7 -u -v *$i.zip; done;
...
echo "------------------------------";
echo "LEFTOVER FILES IN /opt/Hoodlums/";
ls /opt/hoodlums/;

I can run this script inside of my Docker container to make sure that it's catching any edge cases I set up in SetUpEnv.sh without worrying about accidentally touching my local /opt/ dir on my MacBook Pro.

Which is good, because I don't want to accidentally nuke Gradle or Vagrant!

Sometimes I need to run my data through a sanitizer before I can use it to find files.

For example, one of our inputs is 163Killer and a file we're testing against is 0277_Killer.yaml. We need to extract Killer to match that yaml file. That's easy enough with an extra loop and some sed magic:

#!/bin/bash
WHALETYPES="123Fin 456_Gray 495Beluga 385_Livyatan 012_Blue 978Bowhead 149_Humpback 163Killer"
SANITIZEDWHALES=""

function whaleSanitizer {
  input=$1
  input=`echo $input | sed 's/^[0-9]*//'`
  input=`echo $input | sed 's/^_*//'`
  echo $input
}

# B WHALES
for i in $WHALETYPES; do
  SANITIZEDWHALES="$SANITIZEDWHALES $(whaleSanitizer $i)"
done;

echo "------------------------------";
echo "LEFTOVER FILES IN /var/lib/docker/";
ls /var/lib/docker/;

Without turning this into a Bash post, we're running the WHALETYPES var through the whaleSanitizer function and into the fresh SANITIZEDWHALES var. We're then using that new var to find files we need to shred.

Then, at the end of the script, I ls all of the dirs I touched to make sure the files I wanted to get rid of are no longer there.

After I simplified this a bit in order to share it, it ended up being a pretty simple example.

I'm spinning up a container, creating a temporary test environment, then running my cleanup script against that test environment. If it doesn't work right, I can make some changes, set my environment back up quickly, and try again.

An added benefit is that this is easily repeatable. The README.md file with the instructions on setting up and running the test environment is only 176 bytes. That's right. Bytes.

When I inevitably need to update this script, pulling down the GitHub repo will take a few seconds. I can make my changes, and have a container stood up and ready to test against in a matter of minutes. Shorter, if I already have the centos:5.11 image pulled locally.

Hopefully, this makes the benefits of containers seem a little more real. If any of you read this far and have had to spin up containers for a similar kind of project (work or personal) drop it in the comments. Let's get some discussions going and help the newer folks learn when to reach for containers!

Also, as I work on my Compose -> Kubernetes project/article, I'm also waiting on some fun toys to come in from Amazon. This will for sure be its own article, but I'm hopping on the Raspberry Pi Kubernetes cluster bandwagon.

Photo Credit: https://www.c4labs.com/product/8-slot-stackable-cluster-case-raspberry-pi-3b-and-other-single-board-computers-color-options/

Until then, stay frosty.

https://henryneeds.coffee
Blog
LinkedIn
Twitter

The One Where It Turns Out I Misunderstood Kubernetes

Henry Quinn — Fri, 19 Jul 2019 18:15:44 +0000

I just finished Nigel Poulton's The Kubernetes Book. I highly recommend it, even though it pointed out some pretty big misconceptions I had about how some important K8s components work.

It's a great surface level intro to the world of Kubernetes and how it can help build projects that can scale and self heal. Excellent for beginners and intermediate users alike.

I also took some time off from posting. Turns out that DevOps stuff takes a little longer to learn and internalize than I thought. Going to try and scale down to weekly posts and keep this going longer than a month.

Burnout is a really real thing.

I've touched on this in some previous posts, but my time with the U.S. Courts was a lot of trial under fire "learn as you go." In most work environments that wouldn't be a problem, but there isn't much in the way of traditional Sr./Jr. developer relationships within the judiciary.

What that means, in a practical sense, was that I was responsible for teaching myself any technologies that I didn't start the job already knowing.

At a certain point, I was handed a ColdFusion/Informix/MariaDB application and was tasked with getting the application into a certain amount of other districts within a certain amount of time.

For a good laugh before your weekend starts, take a look through this job posting for my old position that just went up. I'll be very surprised if they find a developer they're happy with for those job requirements and salary range.

I think it's telling about the career path I would eventually take that my first thought wasn't about ColdFusion being a garbage language, but rather about how I would both deploy and maintain this application in multiple courts.

Enter: containers.

I had to teach myself Docker, and eventually OpenShift, from scratch. There were a couple of folks working for the Administrative Office in D.C. moonlighting as developer advocates who helped me along the way. I owe the current state of my career to Jonathan, Carl, and Tom; but for the most part, I was learning these complicated technologies in a silo.

I'm unsure if it was real or false competence that made me think I had a handle on everything as we were rushing towards our initial rollout, but I thought knew everything. Typical 20-something bravado.

That is, at least until I read The Kubernetes Book. I learned that even though I had monkey wrenched our application into a deployable state, I had three big misconceptions about how Kubernetes works under the hood.

1.) How services and deployments differ:

When I was getting our JDash product ready for our initial deployment I was mostly just taking known good .yaml files and modifying them to fit my own needs - basic round peg -> round hole stuff.

It was enough to get the job done, but I wasn't understanding what I was doing.

I had a basic understanding that pods were wrapped in containers that can optionally be wrapped in deployments, but didn't have a firm understanding of what these first-class Kubernetes objects provided.

Turns out that there's a base paradigm of separation of concerns that stated that pods should typically (with a few exceptions) only contain one type of container.

Then, those pods can be wrapped in deployments. Those deployments can be configured to keep a certain amount of pods available at all times (including rolling updates) so that your applications gain the ability to scale and self heal.

Thus, the power of containers (mostly easy scaling) in K8s lies in deployments managing pods, rather than the containers themselves.

Understanding that clicked some things into place for me.

2.) How PVs and PVCs work:

I had a loose understanding that PVCs (persistent volume claims) were related to PVs (persistent volumes) in that they both facilitated some kind of persistent data storage to power our ethereal containers.

However, I thought that a PVC was always what provisioned the space a PV took up.

Turns out, I was half right.

Typically, in K8s, you would create a PV that utilized some sort of storage from a storage provider that would then be "claimed" by the PVC. The PVC acts as a ticket that allows a pod to access the storage provisioned in the PV.

Additionally, though, there are K8s resources called storage classes. These set up a connection to a storage provisioner that allow for dynamic storage allocations. That way, you can point a pod's PVC to a storage class and have your PV automatically created and connected to your pod. This behavior works across local solutions (NFS, SMB, etc) and cloud services (AWS, Azure, GCE, etc).

Not having to manually create persistent storage for applications at scale is a HUGE benefit of K8s that I had no solid understanding of.

3.) What exactly labels do:

I had played with labels before, but mainly as adding my name as an author to Dockerfiles.

FROM lucee/lucee5:5.0.1.85

LABEL author="Henry Quinn <henryquinniv@gmail.com>"

# TOMCAT CONFIGS --> OPTIONAL to implement if you need custom Tomcat config.
COPY config/lucee/setenv.sh /usr/local/tomcat/bin/
...

That said, I had no idea what kind of insane power they provide you with an orchestrator like Kubernetes.

Turns out labels are what connect everything. Let's take services for example.

Services are what enable easy networking within a cluster. Short story is that services provide a stable IP address, DNS name, and port. They handle all of the complicated networking bits of forwarding traffic on to ever-changing amounts of pods, containers, and other resources.

That way https://domain.tld always points to your application, regardless of how many pods have been created to accommodate traffic.

But, as your service is handling load balancing for you, it needs to know which node(s) it can throw requests to. You may have a six node cluster with a particular app only running pods on three of those nodes. If a user tries to visit your application, that service is going to be using labels to figure out how to handle it.

That service will be configured to have certain label selectors that will match certain labels you provide to your pods in your yaml.

In short, those labels appear to be what provide most of the magic in Kubernetes. They're what help your cluster master node draw the complicated map across your entire cluster so that it knows how to navigate any kind of request that comes in.

Understanding that is really what made everything click for me.

All in all, I think I was able to get 80% of my way to a solid understanding of Kubernetes on my own.

And I'm damn proud of that.

However, reading this short (160 page) book made the last 20% click into place. And that understanding is going to power a big new project I'm picking up at work.

But more on that later. ;)

Now that I have a better understanding of K8s, I'm going to $ kubectl apply myWeekend.yaml. See y'all next week. Until then, stay frosty.

https://henryneeds.coffee
Blog
LinkedIn
Twitter