DEV Community: Kira McLean

Ethics Are Missing From Software Development

Kira McLean — Wed, 10 Mar 2021 16:14:06 +0000

There is something conspicuously missing from software development education. As developers, we learn all about how to write, organize, design, share, use, and test software, but the discussion of how that software will actually be used and what it's impact on the world will be is usually an afterthought.

What discussions do happen tend to be around licensing and intellectual property, with the main focus on not getting sued. That stuff matters, but it doesn't come close to addressing and grappling with the total impact of the tech industry.

The effects of software on the world are enormous. It affects the way we do everything and shows up in every corner of our lives:

how we keep in touch with friends and family
how we get around
how we work
how we learn about the world and make decisions
how we grow up
how we affect the planet

Preventing our software from making the world worse should be at the top of our minds as technologists, but it's almost always on the backburner. There's a huge disconnect between writing some characters on a computer and seeing effects in the real world.

I'm worried that's a really big problem, though. I really don't think it's an exaggeration to say that some of the most dire problems we need to collectively deal with in my lifetime are caused or exacerbated by software.

Things like the environmental crisis. Bitcoin bullshit already consumes more resources than many countries. Or the mental health crisis. There's good evidence that suggests the internet makes us meaner, angrier, and sadder than ever. Or the erosion of democracy worldwide. There's no doubt that social media is responsible for flooding our information ecosystem with disinformation, which spreads like cancer through our networks and leads us to making irrational decisions that can have deleterious long term effects.

I'm not advocating for every software engineer to become an ethicist, but we can do a lot better at being aware of the impact our work has on the world. It's definitely not nothing, and sometimes it's everything.

Building A Fast Command Line App With Clojure

Kira McLean — Fri, 19 Feb 2021 02:57:34 +0000

Like most developers, I always have about a hundred ideas for little tools or apps I wish existed. Every once in a while I get the time and energy to magic one of them into existence. Clojure is my language of choice these days, but at first glance it's not super well suited to building little command-line apps (which is usually what I start with). Some things that make it not an obvious first choice:

Slow JVM startup time. It usually takes a second or two or more to fire up a JVM, which is an unacceptable startup penalty if your whole app is just a little utility meant to run fast.
No coherent ecosystem. The Clojure community is very averse to batteries-included solutions. There are good reasons why, a main one being that we spend vastly more time maintaining apps than setting them up so we should aggressively avoid including non-essential dependencies, which add to our maintenance burden. This is fair and does work out really well for long-running projects (which is most Clojure projects). But, this means it's often frustrating and slow to get a new project started, compared to some languages at least. There's no clojure new-cli-app type command you can just run to get a new app that works in 1 second.

This post is about how to build a command-line app with Clojure, using tools.deps and GraalVM.

I assume you already have Clojure (including the CLI) and GraalVM installed. I use jabba to manage JVMs on my machine, and installed and setup a GraalVM by running jabba install graalvm-ce-java11@20.3.0 and then jabba use graalvm-ce-java11@20.3. You also need the GraalVM native image utility, which I installed with gu install native-image.

GraalVM and Clojure community to the rescue

GraalVM is basically a super fast JVM, which solves the problem of slow startup time. You can use it to build a standalone executable out of your Clojure app that will run instantly.

Even though there's no batteries-included way to manage Clojure projects, the community has put together a lot of great tools and guides the cover all the bases. The community seems to be converging around the official Clojure CLI and associated tooling as the preferred way to manage Clojure projects. It's extremely well designed, like most things Clojure, but, also like most things Clojure, it's very bare-bones. It's not an all-in-one command-line utility you can use to manage your whole project, like the angular or rails CLIs (which I didn't appreciate nearly enough in my former life 😢). You need to configure the Clojure CLI itself for it to be useful, but luckily that's really straightforward to do. What follows are the steps I did to make a new skeleton command-line app in Clojure. It follows the steps from this great guide, but I included the actual commands here because I use the Clojure CLI (clj) instead of lein to run things.

1. Make a new Clojure project

I use Sean Corfield's clj-new project to initialize new Clojure projects. Install it for your environment according the instructions in his README, then run clj -X:new :template app :name kiramclean/test-cli to generate a new Clojure project (but replace kiramclean/test-cli with <your-name>/<project-name>).

2. Make an uberjar

The app template from clj-new includes a default namespace that just prints "Hello, World!" and an alias for building an uberjar, which is just a java app that includes all the dependencies it needs so it can run on its own without worrying about what's installed or not on the host.

Run clj -X:uberjar in your app directory, which should build a test-cli.jar. You can run your app now like java -jar test-cli.jar, and cry about how slow it is.

3. Make a standalone executable with GraalVM

Now you can use GraalVM to turn your uberjar into a snappy CLI. Run this magic command (note the names -- the -jar option is the location of the uberjar you just made and -H:Name= is the name of your future executable).

native-image --report-unsupported-elements-at-runtime \
             --initialize-at-build-time \
             --no-server \
             -jar test-cli.jar \
             -H:Name=test-cli

It takes a while on my machine for that to finish, but once it does you're good to go! You should have a standalone executable now that you can run from your terminal, which executes your Clojure app natively, and is way faster than running the jar on a regular JVM! Cool.

❯ time java -jar test-cli.jar
Hello, World!
java -jar test-cli.jar  4.31s user 1.10s system 113% cpu 4.792 total

❯ time ./test-cli
Hello, World!
./test-cli  0.05s user 0.01s system 70% cpu 0.086 total

That's all for now

I made an executable bin/build script in my project with this in it to make the two steps above simpler:

#!/bin/bash

echo "Build jar..."

clj -X:uberjar

echo "Nativize it..."

native-image --report-unsupported-elements-at-runtime \
             --initialize-at-build-time \
             --no-server \
             --no-fallback \
             -jar test-cli.jar \
             -H:Name=./test-cli

echo "Success! Good to run ./test-cli"

The next thing I want to do is add some command-line options and a help menu, but this is already getting kind of long, so I'll leave it here for now. Happy coding 🙂

Think of a thing's identity as it's value, not it's container

Kira McLean — Wed, 10 Feb 2021 03:05:52 +0000

Writing code that works is hard. I've been exploring things that make software complex in an attempt to learn how to write better code. I guess "better" is subjective, but in this case I mean "easier to change" and "with fewer bugs". State is one big factor that makes software complex. Most code that does anything useful needs state, though.

How we model that state can make a big difference in how easy or hard it is to understand code. It really comes down to reference vs. value equality semantics, but those are abstract terms and ways of thinking about things that nobody really talks about in real life.

It's easy to gloss over this topic in a conversation about computery things, but it's one that's really worth thinking about a bit because it can have a big impact on our ability to write code that works.

Reference semantics

In most of the popular languages, each object is a unique, distinct thing, regardless of its current state. Take the example of dates in JavaScript:

> date = new Date() // Thu Jan 28 2021 21:54:34 GMT-0500 (EST)
> date.setDate(date.getDate() + 1) // 1611975274726
> date // Fri Jan 29 2021 21:54:34 GMT-0500 (EST)
> date === date // true

We make a new date, then change the internal state of that date, setting its current value to tomorrow, yet we still say it's the same date.

The thing is, it's not the same date. January 28th is not the same as January 29th, but the way we describe objects in most programming languages forces us to wrap our heads around this unnatural way of thinking about what something is.

The flip side is equally confusing -- when you look at things that seem like they should be equal but aren't. Like in JavaScript again:

[1, 2, 3] === [1, 2, 3] // false

You can see that [1, 2, 3] is actually the same as [1, 2, 3], and there are a lot of situations where it would be convenient to be able to check that programatically, but in most languages they're not actually equivalent, because each thing (array of 1, 2, 3, in this case) is a unique entity. The identity of a thing is defined as the container, the object that holds all the attributes. The current value of those attributes does not affect an object's identity in these languages.

Re-inventing equality

Thinking of identity this way forces us to re-invent what it means for things to be equal. In languages that refer to a container of attributes as a thing's identity, we have to re-define what equality means in some domain-specific sense for each new type of object. In order to check whether two obejcts are equal, you have to write extra code to check whether the current value of their attributes are equal. That can add up to a lot of extra code!

It also adds the issue of having to think about time to our code, because in this world whether two things are equal or not depends on the values of their attributes at the instant you check, but that could change. To check whether two things are equal you have to effectively pause the universe and grab a snapshot. All of this adds unnecessary complexity to code. It's unnecessary because it's imposed by the language, not a consequence of logical constraints.

Unwrap your data and think in values instead

We can get rid of this entire category of problem by thinking about a thing's identity as it's value directly, intead of wrapping it in a container and calling that the identity. There are a lot of languages where the equivalent of [1, 2, 3] === [1, 2, 3] is true, as we can intuitively see that it is.

Working with data directly like this is much easier to reason about and removes whole categories of problems to free up your mental bandwidth to work on your actual app, not problems introduced by your language.

You never have to care again about what fields to compare to see if things are equal. This also solves the time problem, because values are inherently immutable. 2 is always 2, it will never be different no matter how much later you check. This means you no longer have to care about when objects change in your code -- if you unwrap the objects and just work with values directly, it suddenly becomes possible to zero in on one tiny piece of your code and stop caring about how all the rest of it might affect the value you're currently trying to do something with.

Reference semantics add unnecessary complexity

The biggest win with writing code this way -- using naked, immutable data structures -- is that it reduces the complexity of your whole codebase. The value of a given thing is always transparent, and never changes over time. This removes whole classes of complexity that we can just stop caring about. You never have to worry again about what other methods might sneakily change the thing you're working with. And you don't have to care about when things happen anymore, either. You can always know that whatever value you're dealing with is static for the purposes of your current context.

Clojure is one language that is designed like this by default, but it's possble to write code this way in many languages. Give it a try and see what you think!

How state makes software complex

Kira McLean — Sat, 30 Jan 2021 21:07:27 +0000

What makes large codebases so hard to reason about, and so hard to extend? I really want to find out because I'm tired of losing my mind working on them. One paper I love called "Out of the Tar Pit", by Ben Moseley and Peter Marks, argues that state is the main source of complexity in large software systems. This article is an overview of a couple of the ideas from that paper, along with some discussion about how we might conceptually think about necessary state in software in order to improve its overall quality.

How does state cause complexity?

The main issue with maintaining state in a program is that it quickly leads to a proliferation of different internal conditions that can affect the output. The more state there is, the more possible combinations it can be in, the harder it is to predict and wrap our heads around what the whole system is going to do with a given input. Every new piece of state introduced into a system dramatically increases the total number of possible states the whole thing can have, and this number quickly exceeds the capacity of our brains to conceptualize it all at once, which leads us to write code that behaves in unexpected ways. Bugs!

When the output of a program depends not only on its inputs, but also on its current internal state, it is inherently unreliable because we cannot guarantee that the inputs we're giving it will always produce the same result. This is how we end up with systems that behave in unexpected ways. Most programs need at least some state, though, so how can we deal with it?

Hiding state does not count as dealing with it

One classical approach to managing state is encapsulation -- breaking state down into small pieces and putting different parts inside different classes or objects. As Rich Hickey says, though, putting the mess into a container does not fix the problem. It just means you're in charge of that mess now.

Remember the fundamental problem causing complexity is that stateful programs have too many possible internal states which affect their output in ways that are opaque and therefore difficult to understand. Separating the state into pieces and regulating access to it through methods doesn't solve that. There is nothing to stop multiple methods from accessing the same bit of state, or even multiple different objects calling those methods and manipulating the internal state of some other object. With this approach, the behaviour of the program as a whole still ultimately depends on the current internal state of its components.

Most languages do support things like private methods and attributes and encourage best practices like loose coupling, but as Murphy's law says, "anything that can go wrong will". Whatever ways there are in a given language to manipulate the internal state of objects will eventually be used. Good practices are never enough to save us from ourselves.

Ignoring state does not count as dealing with it

Another classical approach to managing state is basically to reject it -- insisting that useful programs can be modeled as pure (mathematical) functions, i.e. ones where the only thing affecting the output is the input. This is a nice way to think about computation in many ways, but at the end of the day it's not pragmatic. It ultimately just kicks the can down the road.

The vast majority of programs/systems/apps exist to be used by people, and those people are not all identical, so many of them want to use those systems in subtly different ways. This is one reason why most programs need to deal with at least some state.

One typical approach to managing user input or other essential state in programs made mostly (or only) of functions is to add arguments to all the functions that would normally store or access the internal state of the program, effectively passing some initial state to the entry point of the system and threading that state all the way through.

In many such programs these extra arguments end up being large compound values, acting effectively as a single pool of global variables, which in the end can be just as confusing. This is still a huge improvement over the encapsulation approach because at least each individual function can be more easily understood in isolation. It is ultimately simpler to understand something when all of its inputs are explicit and transparent. But, if one of those inputs is a huge map of stuff where only a slice is actually relevant, we're not a whole lot better off in the end.

Hidden, implicit, mutable state is a major source of complexity

These two approaches to dealing with state reveal some characteristics of the kinds of state that are fundamentally bad to have in a system (because they increase its complexity, which makes it harder to reason about, which makes it less reliable).

Hidden (or implicit) state makes programs more complex because we have to do extra work to figure out what the output is going to be. The mental overhead of figuring out how this hidden state affects the output can quickly become overwhelming and make systems impossible to understand.
Mutable state makes programs more complex because we cannot guarantee the state will be what we expect at any given time. We cannot trust best practices alone to isolate state mutation to only the right places, so we can never be certain what the values of mutable properties are going to be. This also means we have to pause the universe in order to read mutable state, which fundamentally breaks out model time in programs that have it. But that's a topic for another day.

Make state explicit and immutable to make less complex software

There are obviously loads of things that make programs complex. But state is a big one, specifically hidden, implicit, mutable state.

In any system where the outcome of running some code depends on the internal state of the system, you have to know the state of that system at the moment the code runs in order to know what the output will be. And the more parts of a system you have to pull in to your mental model just to understand how one small piece works, the harder it is to write code that works.

To write less complex (and therefore more reliable) software, make implicit state explicit and use immutable data structures. This minimizes the number of things you have to account for when looking at a single method/function, which makes it easier to write better software.

I daydream about a world where most software is written in languages that enforce these constraints by default, but in the meantime there are libraries in many popular programming languages (like JavaScript, Java, and Python) that can help.

Becoming A Software Developer

Kira McLean — Tue, 19 Jan 2021 15:25:54 +0000

I get asked a lot for advice on becoming a software developer. I didn't study anything related to software in university, but now it's what I do for a living. I also spent about 3.5 years mentoring with Lighthouse Labs, a local web development bootcamp, and watched (and would like to believe at least in some cases, helped) hundreds of people of every age from wildly diverse backgrounds get jobs as software developers.

This is my advice for people with no technical experience who want to land their first job writing software. It's based on my experience as both a person who did it myself and as a mentor to other people who did it. There's no single right way to go about this, and no matter how you do it's a lot of work. But these are some things that have worked for other people, myself included. These suggestions are arranged roughly in order, but of course many overlap and kind of happen at the same time.

Know what you're getting yourself into

Software engineering is nothing like I expected it to be. All the hard parts involve dealing with people. Part of my original motivation for choosing a career in software was thinking I wouldn't have to deal much with people. I wasn't very good at it and didn't enjoy it, but my fantasy of just sitting in front of my computer all day and watching my bank account fill up was totally delusional.

The reality of writing code for a living, in my experience, includes dealing with a lot of office politics, entitled middle managers, stressed out co-workers, and yes, condescencion and occasional harassment (especially if you're not male). I find the culture of the tech industry generally toxic and abusive and it has very nearly broken my spirit on multiple occasions.

You won't be alone dealing with this. Your co-workers and countless people across the industry share the experience. And it is ultimately bearable.

I don't say this to discourage anyone from getting into software, but just to highlight that the stuff you hear from recruiters is bullshit. It's not all free trips and fancy perks and big paycheques. It is those things, sure, but the reality is much less glamorous than they make it sound.

Being a software developer means getting paid to solve other people's problems. The rest is fluff. This makes it a solid career path, though, because other people have a lot of problems.

It is still an empowering and lucrative career. Despite my generally negative view of the industry, I still highly recommend and help people into a career in software regularly. Most individual software developers, like most people, are genuine and kind. And I can't speak to problems in other industries first hand, but I gather most high-paying, male-dominated industries are pretty similar anyway.

Meet lots of people

I was referred to three of the four programming jobs I've had so far through friends. I made friends with those people who were already programmers at meetups. I didn't ask them to get me a job, but a lot of companies are pretty much always looking for new software engineers, so when your friend hears you're looking for that kind of job, they can bring it up at work. I still did interviews, but it makes an enormous difference when someone the boss already knows and trusts can vouch for you.

A word on meeting strangers. I, like many people who are into nerd things like software, am the most introverted person I know. I found hanging out in groups of new people excruciating at first, but honestly learning how to not be awkward around strangers is just as important as learning how to code if you want to do well in your career. I just went to the meetups and stood off to the side, awkwardly making eye contact with a few random people. Eventually a friendly extrovert would come up and initiate a conversation, and before I knew it I could hold my own in a crowd.

Check out meetups for programming languages you're interested in, or if there aren't those kinds of groups where you live, join online communities. Most programming languages have a slack or discord server where people hang out, and in my experience people are usually very welcoming and kind to newbies (at least in the Ruby, Elixir, and Clojure communities; all of which I highly recommend).

Learn how to code, in public

Without any formal education in software development, you have to offer something else to prove you actually know how to write code. I did this by building a bunch of crappy side projects. The very first thing I ever did was replicate the Google landing page with HTML and CSS. The first actual app I made was a blackjack game with Sinatra. I also made Twitter and Reddit clones with Rails, and little browser games like Pacman with different JavaScript frameworks. Some of that code still exists in my GitHub graveyard if you want to see what I mean by crappy. Other common toy projects include link shorteners, photo galleries, e-commerce stores, time trackers, or clones of whatever software you normally use, like spreadsheets, workflow management, calendars, etc. The possibilities are endless.

Look for tutorials online that walk through how to build the kinds of things you're interested in. If you don't know where to start, The Odin Project and Free Code Camp are two free programs that offer very realistic learning roadmaps. I did parts of both of those but never finished either. I also did several free courses on Coursera and edX, which are both amazing but you have to be careful not to end up in a rabbit hole. Those two mostly have actual university courses, which are super interesting, but not very relevant to things you'll do in a real job.

If your goal is to get from nothing to making money as a developer as fast as possible I recommend sticking to a more targeted learning roadmap like the two above. There are also countless paid ones which vary wildly in price and quality. Be weary about shelling out huge amounts of money for bootcamps. I did pay for a few months of Launch School (which at the time was called "Tea Leaf Academy"), and it was the best career move I ever made. I highly recommend them because they're in a sweet spot of offering a robust and relevant curriculum without costing a fortune. It's not free ($200/month), but a few hundred dollars is a very small investment in the scheme of your career. Apparently they also offer a deferred payment option now, too.

The career-oriented bootcamps will also teach you loads of useful practical things they don't teach in the free university courses, like version control (git), testing, and how to collaborate with other developers on one project.

It took me 6 months of studying, meeting people, and building crappy software full time before I got my first internship. 95% of Lighthouse Labs graduates are employed as software developers in that timeframe. Launch School delivers impressive, concrete results for many of their students, too. I never finished most of the courses or programs I started. I funded that time off by living like a peasant on money I made running one of those student painting businesses the previous summer. Learning to code can feel like a slog at times, but just stick with it and you're very likely to be among the vast majority who succeed at this self-educated developer thing. It is a totally achievable and realistic goal and you're in good company.

Other than doing free online courses and building apps from tutorials, other ways I learned in public during this time included blogging about my learning journey, making myself a portfolio website to showcase my little projects, and volunteering with some local tech communities. Do what works for you, but focus on ending up with at least a few medium sized projects you can show someone to prove you know how to make software that works.

Learn best practices

A surprising number of experienced developers still write really horrible code. Spend some time learning about software design and software engineering best practices to avoid becoming one of them. How you do that depends a lot on how you learn. (BTW, you should figure that out before you embark upon a journey of self-re-education for a career change). I absorb a lot from reading and prefer to learn by studying theory over trial and error, so reading and watching talks works really well for me. YMMV and you do ultimately just need experience to really get this stuff, but you don't have to start from scratch.

Read "The Pragmatic Programmer" by Dave Thomas and Andrew Hunt, and anything by Sandi Metz. Watch Rich Hickey's talks, especially "Simple Made Easy", and Sarah Mei's "Livable Code". Other classics include Eric Evans' "Domain Driven Design" and Martin Fowler's "Refactoring". There are countless others and everyone has their opinion about what should be included in the software engineering canon, but these are my recommendations. They'll provide a solid foundation and ample fodder for further reading and watching.

Pour your heart out

Once you can write code that mostly works, start looking for a job. Put the word out that you're seeking your first role writing software. Hopefully one of your programmer friends will know about an open position, but you can still apply to random jobs on the internet, too. This has a very low success rate, but it's possible. I got one of my four jobs by firing my resume into the abyss. I included a very long and heartfelt cover letter about how I loved programming so much and really wanted to work for this specific company (which I did), and how I learn fast and really believed I could do the job if they'd give me a chance.

It worked. The hiring manager said my letter caught his attention. I had one year of experience at that time and had never worked remotely before, but I got the job.

Don't be afraid to be authentic. Don't lie and say you know more than you do, but don't sell yourself short, either. Bootstrapping a career in software is no small feat and there are many teams out there who would love to hire the kinds of self-starting fast learners who do it.

Don't be too picky (at first!)

The reality is that your first job or two will probably be pretty crappy and underpaid. At this point you're trying to convince someone to take a chance on you, and it's a trade off between your dignity and your future career. In the very beginning, you should prioritize gaining experience over almost everything else. Obviously never put up with anything you're uncomfortable with, but if you can stick it out it's worth setting the bar pretty low just to get your first few months of real job experience, IMHO.

I lucked out and worked with the most amazing team in the world for my first job as a developer, but I discovered from the students I mentored that that is very much not the norm.

There are pros and cons to every kind of company, so don't worry too much about finding the perfect fit right away. Focus on gaining experience at first so you can have the freedom to focus on finding the perfect fit later. I've worked with a few different kinds of companies and there are pros and cons no matter where you go.

Big corporations usually have a lot of experienced developers and lethargic processes, so there's more opportunity to take your time and get help from people more advanced than you. The pressure is often pretty low because people have such low expectations for new developers, which honestly is fair. You have a lot of room to screw up because you're such a tiny cog in a massive machine. The downsides depend on your personality. For me it was that I find most large tech companies to be evil and didn't like the soul-crushing feeling of helping to build something I hated.

Startups usually have fewer people available to be mentors, but more opportunity to take on disproportionate responsibility for your level, which can be an amazing learning experience in a different way. You also have a lot of room to screw up, but it's more because your screw ups just blend in with the overall culture of moving fast and breaking things. I now believe that's a ridiculous way to run a business, but when there's widespread acceptance of breaking even important things, you have a lot freedom to experiment and learn by trial and error.

Be a good student

Lastly, once you finally land your first job, be a good student. Treat your first year or two on the job like paid education, not your chance to go in and show everyone how it's done. You know way less than you think, and you have no idea what it takes to build functioning software systems, let alone to deploy or maintain them. Always be humble and eager to learn. Soak up every minute of attention you get from more experienced developers around you and learn from them. Not everything they say is wise or true, but pay attention and copy the things that work whilst ignoring the things that clearly don't.

Learn how to ask good questions. Don't be afraid to ask for help, but before you do, at least try to figure out the problem yourself. Write down what you tried, and explain why you're surprised your attempt didn't work.

There are certain things you can only really learn from experience on a real world project, so see what opportunities there are to learn those things, for example:

securing and backing up actual important customer data
deploying a huge app with no downtime
refactoring large, messy, legacy codebases
adding new features to large, messy, legacy codebases
fixing obscure bugs with no instructions to reproduce them

These are just a few examples of things you won't run into on your little demo projects. Pay attention to the problems you're dealing with at work that you don't run into on side projects and learn how to fix those. That's your pathway to levelling up. Most importantly, never stop learning.

That's all

Software development is a great career. It's allowed me to be financially independent and stable, travel the world, meet cool people, and lots more. My path won't work for everyone. There are as many different ways to become a software developer as there are people, but I've seen countless aspiring software engineers go through their own journeys and many of them touch on these points, so see how they fit in to your own path. And let me know how it goes 🙂

What I Use Now Instead Of Google

Kira McLean — Wed, 06 Jan 2021 04:43:07 +0000

I made a goal for myself in January 2020 to stop using Google products by the end of the year. That might sound like way too generous a timeline, but Google owned pretty much all of my data at that point, so it was a fairly large project. Plus I'm a slow and steady kind of person. I know if I give myself a generous enough timeline I can accomplish even things that seem too hard for me at first.

First, Why

Since I got into programming about 5 years ago, I kept hearing all these bad things about Google and how horrible of a company it is from other tech people. It always seemed a bit exaggerated to me, but the evidence has been piling up over the years. Learning about their involvement with the US military's Algorithmic Warfare Cross-Functional Team (Project Maven) was the last straw for me ¹, but even before I learned about that I was pretty uncomfortable with a lot of what I heard. In Google's defence they eventually did pull out of the project after massive backlash by thousands of employees.

Still, I'm pretty convinced now that their continued existence is a catastrophe not only for public safety, but also for the environment, gender equity, the economy, fair labour practices, privacy, journalism, democracy, race relations, and the project of civilization itself. But anyway, the point of this post isn't to motivate you to also quit Google. I'll tell you how I really feel some other time.

The rest of this post is about the tools and services I replaced all the Google things with. I did mostly accomplish my goal, with a few caveats which I describe in the relevant sections.

Replacements

Here's the short version:

GMail → ProtonMail
Chrome → Firefox developer edition
Google search → DuckDuckGo
Google Drive → Sync and Backblaze
Google DNS → Cloudflare DNS
Maps → Apple maps
YouTube → Netflix, TED talks, conference archives, and still a little YouTube (anonymously in a Firefox container)
Google Analytics → Fathom
Everything else (calendar, reminders, photos, docs, video chat, news feeds) → Nextcloud, running on my own instance

And here's the long version:

GMail → ProtonMail

ProtonMail has free accounts, but I pay for the lowest level that allows use of a custom domain with it (€48/year) so I don't have to change my email address ever again if I want to switch providers. It was quite a pain to change my email address all over the place, but I started in May and just did it slowly over time. I highly recommend decoupling your email address from your email provider if you're considering switching. First of all an email from a custom domain seems more, not less, serious than a gmail address to most people. But the main reason is just so you don't have to change your email address ever again if (when) you want to switch to some newer, better email provider.

I set up forwarding from my old GMail account to my new email address then filtered my mail for anything sent to the old address. If it was someone or some place I wanted to continue hearing from, I updated my email address with them. If not I unsubscribed. This turned out to be a wonderful opportunity to purge my newsletter and other email subscriptions. Also I'm happy to report most companies are now (finally) respecting unsubscribe requests. Two (looking at you Geektastic and Rakuten) continued to spam me after I requested they stop, so now I filter out all their mail as spam.

Chrome → Firefox developer edition

I primarily use Firefox developer edition now for my browser, including for work. The developer tools are just as good as Chrome's for the kind of work I do.

There are some web apps that literally or effectively only work in Chrome, which ironically perfectly illustrates the impetus for this whole undertaking. For those I use this ungoogled Chromium browser, installed via homebrew.

I still have Chrome installed on my computer because I need it for some work things. We use chromedriver for some integration tests and it wasn't trivially easy to trick it into using my Chromium installation instead of looking for Chrome. Replacing chromedriver is a headache, and also not my call to make at work. But it's also not really the point for me. If Google was reduced to a browser that developers can easily launch and control programmatically, I'd be satisfied. I don't use it for anything other than running automated tests at work now.

Google search → DuckDuckGo

I mostly love DuckDuckGo, but there are certain categories of searches where Google returns better results. DuckDuckGo doesn't seem to return many results from forums (like stack overflow or other stack exchange sites), which is where there the answers to a lot of my questions are, unfortunately. So I still use Google Search sometimes, but I do it in a Firefox container without being logged in to a Google account, which at least helps to avoid some of Google's incessant stalking.

DuckDuckGo does seem to return better results when the answer is an image, video, or regular web page, which is cool, and I suspect means it would be a perfectly fine replacement for most people. For example this blog is easier to find via DuckDuckGo than Google, which is impressive because I share a name with a small-time TV celebrity who typically dominates search results. We even look sort of similar, it's wild.

Google Drive → Sync and Backblaze

I use Sync to make my most used files available anywhere, and I've started moving a huge backlog of documents and notes I don't need to access often but don't want to throw away to Backblaze for longer term storage, just because it's cheaper. So far I just use their web UI to upload things in bulk and browse my files, but I'm looking into ways to do that more efficiently. I really need to de-clutter my digital life. I'm a major hoarder when it comes to digital files. I have scarcely deleted a document in the last 10 years, it's getting a bit ridiculous. But that's a goal for another year 🙂. For now I'm just putting all those files I don't really need to access but don't want to throw away in Backblaze to deal with later.

Between Sync and Backblaze I got 15GB of free storage, which is plenty for now, though as I move more and more or my scattered files over I'll have to start paying. Backblaze offers some of the cheapest object storage there is, at least.

DNS → Cloudflare DNS

I used to use Google's DNS servers (8.8.8.8 and 8.8.4.4), now I use Cloudflare's (1.1.1.1 and 1.0.0.1).

Maps, YouTube

These ones are harder to replace. I use Apple maps now most of the time, but I still use Waze (which was acquired by Google) for directions sometimes if I'm driving somewhere. For media I mostly watch Netflix, but I also watch a lot of TED talks. I used to also watch conference videos on YouTube. Now I check conference websites for those, and a lot of them have the recordings, although they're often hosted on YouTube anyway. And I also do still use YouTube sometimes, mostly for home workout videos, but also anonymously in an isolated container.

Analytics → Fathom

It's been a while since I used Google Analytics, but I wanted to set up a basic hit counter for this blog and found Fathom. They provide simple privacy-focused analytics that work well enough for me. One really useful feature they have that I consider necessary now is being able to log analytics with a custom domain. Without this, visitors using ad-blockers don't get counted, which most estimates figure is now something like 40% of people, and probably more among tech-type people, like the ones most likely to find this blog.

I tried Cloudflare's new server-side analytics for a month, but the data didn't make as much sense (it showed the overwhelming number of visits to my homepage, even though the other analytics I had set up showed them going to one post that did well on Hacker News, which makes way more sense). Anyway, the numbers didn't seem to add up. I guess to be fair I don't quite understand how Fathom's numbers add up yet either, but their support has been helpful checking it out with me to try to help me reach some interpretation that makes sense. Fathom's dashboard is really simple, at least, which I like because it's easy to understand.

Calendar, reminders, photos, docs, video chat, news feeds → Nextcloud

For everything else I use Nextcloud now. I run my own instance of it, but there are lots of providers where you can just sign up for a simple account like anything else if you're not an insufferable nerd who enjoys maintaining servers, like me.

This is where I have another caveat to mention. I haven't migrated all of my photos off of Google Photos yet because I have about 50,000 of them and it's just a really slow process. I estimated it would take me something like 70 hours to move them all manually, so I'm looking into better ways to do it. I'm confident I can find or maybe build something easier than manually downloading and uploading 50k files in less than 70 hours.

Nextcloud does the job of automatically syncing photos from my phone, though, which is all I needed to be able to delete Google Photos from my devices. I'm not sure I'll actually stick with Nextcloud for photos in the long run. The gallery is a bit lacking. It looks like Piwigo might be a good alternative. Either way, I'm actually storing my photos in Backblaze, so I'll continue looking around for a different client for a bucket full of photos next year as I work on slowly culling my photos collection and migrating it to a new home.

Total Cost

I feel like I've won a lot by doing all this. I own my data now, so Google can't arbitrarily take it away from me, which gives me peace of mind. I'm no longer part of their ad ecosystem, being tracked all around the internet and having my attention sold to the highest bidder. And I also feel good about "voting with my feet", so to speak. The fewer people use Google's free products, the lower their ability to sustain their unethical business model based on selling mass surveillance data.

But I also lost some. Specifically a few hundred dollars.

Getting paid and paying for things in multiple currencies is the bane of my financial existence, but overall the final total comes out to something like CAD$400/year (roughly $300), broken down like this:

ProtonMail - €48/year (CAD$75)
Custom domains (for email and my cloud) - $19.74/year (CAD$25)
Linode server (for my Nextcloud instance) - $84/year (CAD$105)
Carbon offsets - CAD$21/year
Fathom analytics - $140/year (CAD$180)

As you can see, $140 of the total is for Fathom analytics, which an average person probably doesn't need. Without that it comes closer to about $160, which I think is a fair price to pay for privacy, freedom, and ownership. I'm also not paying anything for storage yet, though, so I expect this to cost more next year once I finally get all my photos and archives migrated to Backblaze or whatever I end up using.

I should also emphasize it's entirely possible to switch off of Google to free alternatives. All the services I use have free tiers which are quite generous and most likely plenty for an average user. I do think people who can afford to should support independent software companies so they can run sustainable businesses off the money their customers give them and not off ad revenue, but when money is tight there are still lots of options.

Conclusion

Once I got over the initial mental hurdle of expecting everything to be free or as cheap as possible, it seemed totally reasonable to me to pay that much for all I'm getting. What made it click for me is that all those things aren't actually free. Google sells our information and our attention to the highest bidder to generate profit for themselves, then they don't share any of it with us, despite the fact that their entire business model couldn't even exist if we all just refused to fuel it.

I know I'm just dreaming waiting for the day Google realizes the error of its ways and starts running an ethical business, but that doesn't mean it's not worth taking a stand in the meantime. Individually refusing to fuel the databases and algorithms Google profits from is one tiny thing it's totally possible to do.

Footnotes

1. That previous link goes to a memo saying that the primary purpose of the project is "to field technology to augment or automate Processing, Exploitation, and Dissemination (PED) for tactical Unmanned Aerial System (UAS)", which means making AI for drones so they can better target humans and other "targets". That really bothers me. ↵

How To Set Up Codecov For a Clojure Deps Project

Kira McLean — Thu, 31 Dec 2020 14:23:56 +0000

I recently set up code coverage reporting with Codecov for a Clojure project of mine that uses tools.deps and builds on CircleCI. It turned out to be pretty easy but the documentation for the various parts was a bit ambiguous, so I wrote down the steps here in case you're looking to do the same. You can see all the changes it took together in context in this commit where I set it up for my project.

Assumptions

These instructions assume your project uses the same specific set of tools as me:

Clojure with tools.deps
Github
CircleCI
Codecov

The same idea should work fine with other combinations of services, but you might have to do things slightly differently or tweak some syntax if you use e.g. Gitlab instead of Github or Travis instead of CircleCI.

1. Set up the project in Codecov

Sign in to Codecov with Github
Click "Add New Repository" on the dashboard for your github user/organization (the url should be something like https://codecov.io/gh/<gh-username>)
Select the repo

This should leave you at a screen that shows you a Codecov token. If not, click on the "Settings" tab for your project in Codecov to see thee token. Copy it.

2. Add the Codecov token to CircleCI

In CircleCI "Project Settings" > "Environment Variables" set CODECOV_TOKEN to the value copied from above

I'm assuming your project is already building on CircleCI. If not you can do similar steps to above on Circle (sign in with Github and add your project), and crib my config for a Clojure deps project to get started.

3. Generate Codecov reports from test runs in CI

Add a new alias to your deps project to run tests and generate the Codecov reports using cloverage:

:coverage {:extra-paths ["test"]
           :extra-deps {cloverage/cloverage {:mvn/version "1.2.1"}}
           :main-opts ["-m" "cloverage.coverage" "-p" "src" "-s" "test" "--codecov"]}

Update your run tests step in your CircleCI config to use this new alias:

- run:
    name: Run tests
    command: clojure -M:coverage

4. Send the coverage reports to Codecov

Add a step in your CircleCI config to send the reports to Codecov:

- run:
    name: Send test coverage to Codecov
    command: bash <(curl -s https://codecov.io/bash)

Caveats

Code coverage isn't a great metric for the quality of your test suite. It's pretty easy to write a couple of tests that exercise most of your code but don't prove much about it's correctness, at least in Clojure where your whole program is basically a pipeline to transform data. I still use this metric though. I find it useful as a high level check to see if there are entirely untested functions, which can also help find dead code. Anyway, just a note of caution to not lean too heavily on code coverage metrics as a measure of the quality of your test suite. It's more like a flag that gets raised if you forget to test something altogether. High code coverage doesn't necessarily indicate a good test suite, but low code coverage definitely indicates a bad one.

How To Set Up Your Own Nextcloud Server

Kira McLean — Sun, 20 Dec 2020 00:22:58 +0000

One of my goals for 2020 was to stop using Google's products. There are a lot of reasons why, but that's not the point of this post. I found out about Nextcloud last month and it turns out it's a great replacement for a lot of Google. I don't actually use all of its features, but I've migrated my calendar, reminders, contacts, bookmarks, video calls, photos, and news feeds and I'm really happy with it so far.

There are a lot of companies that will host Nextcloud for you where you just sign up for an account like anything else, but in case you're interested in hosting Nextcloud for yourself this post is basically a brain dump of how I did that. It took me a while to cobble together all the pieces I needed to get everything working from end to end, so I'm hoping this might save someone else from having to do the same. If you know your way around servers and Nextcloud already you can just skim the headings like a checklist to make sure you don't forget an important step. But if you want a succinct overview of the actual steps I did and commands I ran, each section contains those details. By the end you'll see how I installed Nextcloud on my own server, secured it, set up backups, and set up external storage for my photos.

Some parts are pieced together from other partial guides or longer blog posts, so where relevant the references lead to those sources. This post is more of a quick start with just the essential steps. Anything in <pointy-brackets> is meant to be replaced. So the actual command I ran was e.g. adduser kira, not adduser <name>.

What this list assumes you already have

A domain name. I got mine from Namecheap.
An account with a cloud server provider. I use Linode.
An account with Backblaze
An account with Healthchecks.io
Your ssh key
$75/year. I pay $5/month for the server I use, $5/year for the domain name, and $10/year for carbon offsets.

1. Set up a server running Ubuntu and set up ssh access for yourself

Spin up a new server with your cloud provider. I use a "nanode", the smallest server available from Linode, with 1GB of RAM and 25GB of storage, which is plenty more than Nextcloud's minimum specs.
Select an operating system that can install snap packages. I'm using Ubuntu 20.04 (LTS). Nextcloud recommends at least either Ubuntu 18.04 LTS or Red Hat Enterprise Linux 7.
Add your ssh key to the server. There should be a way to do this through the UI where you manage your new server.
Copy the IP address of your new server

1.1 ssh into your server as the root user and make yourself a new sudo user that can also ssh into the machine ¹

ssh root@<server-ip-address>
adduser <name>
usermod -aG sudo <name>
rsync --archive --chown=<name>:<name> ~/.ssh /home/<name>

Close this ssh session and log in as your new user to make sure it works:

logout
ssh <name>@<server-ip-address>

Leave this ssh session open. The rest of the commands below are meant to be run on your server, unless otherwise stated.

2. Point your new server to your custom domain

There should be a way to do this in the admin section for your server. On Linode's there's a "Domains" section in the left admin menu. From there I clicked "Add a Domain" in the top right, then filled in the domain name, my email address, and selected "Insert default records from one of my Linodes" from the "Insert Defaults Records" dropdown, then I selected my new Nextcloud server from the list of Linodes. The steps might be slightly different depending what cloud server provider you're using. By the end you need DNS records pointing your domain name to your Nextcloud server. If you did this through Linode (or whatever you're using), you'll also need to update the nameservers with your domain registar.

3. Set up a basic firewall

sudo ufw allow OpenSSH
sudo ufw allow https
sudo ufw allow http
sudo ufw enable

4. Install Nextcloud

sudo snap install nextcloud
sudo nextcloud.manual-install <username> <password>

5. Set up your domain, enable https, and install an auto-updating certificate from lets encrypt

sudo nextcloud.occ config:system:set trusted_domains 1 --value=<your-domain.name>
sudo nextcloud.enable-https lets-encrypt

6. Enable 2FA on your Nextcloud account ^*

Log in to your new personal cloud at the domain you configured using the username and password you chose above and install the 2FA app
- Click on your initial in the top right corner of the Nextcloud dashboard and select "Apps"
- In the left side bar click on "Security", then search for the "Two-Factor TOTP Provider" app
- Click "Download and enable"
Set up 2FA with this newly installed app
- Click on your initial in the top right corner and select "Settings"
- In the left sidebar, click on "Security" (in the "Personal" section) then check the "Enable TOTP" box and follow the instructions to set up 2FA

I managed to forget my password in the time between installing Nextcloud and trying to log in for the first time. If that happens to you, you can reset it by running sudo nextcloud.occ user:resetpassword <username>.

* Doing this means you will be required to generate "app passwords" in order to log in to your Nextcloud account in third party apps or other devices (to use Nextcloud to sync your calendar or reminders to your phone, for example.) There's a tiny box with a button that says "Create new app password" at the bottom of the "Security" admin section (under "Personal", not "Administration") where you can do that.

7. Set up backups

7.1 Turn on "local" backups

Enable backups for your whole server for a first layer of backups. I did this when I was setting up my Linode (there was a checkbox in the "Optional Add-ons" section for it). Otherwise there's a "Backups" tab in the admin section where you can turn them on. Linode charges $2/months for this.

7.2 Set up "offsite" backups

Install and set up backblaze
- Make a bucket in backblaze for your backups
- Make an app key with access to your backup bucket
- Get the backblaze cli and configure it
- sudo apt install python3-pip
- sudo pip3 install b2
- sudo b2 authorize_account <keyID>
- Copy the key secret from the app key you just made to authorize the Backblaze cli
Create a new user to run the backups and disable password access for it, for security ²
- sudo adduser ncbackup
- sudo usermod -s /sbin/nologin ncbackup ^*
Create directories for the backups and logs
- sudo mkdir -p /home/ncbackup/backups/logs
Create the backup script and make it runnable ^**
- sudo touch /usr/sbin/ncbackup.sh
- sudo chmod +x /usr/sbin/ncbackup.sh
- sudo vim /usr/sbin/ncbackup.sh and copy the contents of the backup script below into your new file, or write your own that accomplishes the same things: ^***

#!/bin/bash
set -e

DATE=$(date '+%Y-%m-%d')

# Output to a logfile
exec &> /home/ncbackup/backups/logs/${DATE}.txt

# Export all your config and data from Nextcloud
echo "Starting Nextcloud export..."
nextcloud.export
echo "Export complete"

# Compress backed up folder
echo "Compressing backup..."
tar -zcf /home/ncbackup/backups/${DATE}.tar.gz -C /var/snap/nextcloud/common/backups/ .
echo "Nextcloud backup successfully compressed to /home/ncbackup/backups"

# Remove uncompressed backup data
rm -rf /var/snap/nextcloud/common/backups/*

# Remove backups and logs older than 5 days
echo "Removing backups older than 5 days..."
find /home/ncbackup/backups -type f -mtime +5 -delete
find /home/ncbackup/backups/logs -type f -mtime +5 -delete

# Keep 14 days of backups in backblaze
echo "Uploading to backblaze..."
b2 sync --keepDays 14 --replaceNewer /home/ncbackup/backups b2://<your-bucket-name>
echo "Nextcloud backup completed successfully"

Let the ncbackup user run the backup script as the root user
- sudo visudo
- Copy this to the end of the file that opens:

# Allow ncbackup to run script as sudo
ncbackup ALL=(ALL) NOPASSWD: /usr/sbin/ncbackup.sh

* If you want to undo this for some reason you can run sudo usermod -s /bin/bash ncbackup

** Note this means you will have 6 copies of all your data on your server all the time -- 5 backups and the live versions. The backups are compressed, but it can still add up to a lot of space. Keep an eye on how much storage your server is using. Running it out of space will probably be one of the first issues you run into. I explain how to get notified when that's close to happening at the end.

*** You don't have to use vim here. Your server probably has nano installed or you can install the editor of your choice. To change the default editor on your server, run sudo update-alternatives --config editor, and choose the one you want.

8. Schedule and monitor your backups

Make yourself a healthcheck endpoint at healthchecks.io and copy the ping url
sudo crontab -u ncbackup -e
Copy this to the bottom of the file: 0 2 * * * sudo /usr/sbin/ncbackup.sh && curl -fsS -m 10 --retry 5 -o /dev/null <your-ping-url>

This will run your backups once per day at 2am (in your server's timezone, probably UTC), but you can set whatever time and frequency you want, just remember to update your healthcheck to match.

9. Test your backups

Backups are only useful if you can use them to restore your data. Make sure yours work before you need them.

To test your entire server backups you can just try restoring the whole server using Linode's (or whoever's) UI. Testing the archived backups we uploaded to backblaze is a little more involved but you'll be glad you know how to do it when you need it.

Repeat steps 1-5, except you can just update the records for your domain that's already set up to point to your new server's IP address(es).
Download one of your backups
Copy the backup onto your new server. Run this in a terminal on your machine (not in an ssh session with a remote server):
- scp /local/path/to/your/backup/ <user>@<new-server-ip-address>:~

ssh into your new server for the rest of these commands

Unzip, rename, and move the backup to a place where the Nextcloud snap installation will be able to access it, then make the root user the owner
- tar -xvzf <backup-name>.tar.gz
- sudo mv <backup-data-dir>/ /var/snap/nextcloud/current/
- sudo chown -R root:root /var/snap/nextcloud/current/<backup-data-dir>/
Import your data
- sudo nextcloud.import /var/snap/nextcloud/current/<backup-data-dir>/
Once it's done, clean up the backup archive
- rm <backup-name>.tar.gz

This should be all you need to restore your Nextcloud installation. It might take a while for the DNS records to propagate, so if you want to test that your restored cloud is working in the meantime you can check it directly at its IP address if you add that to the list of trusted domains:

sudo nextcloud.occ config:system:set trusted_domains 2 --value=<new-server-ip-address>

Note this will only be available over http, so you might get a dramatic warning about security when you visit the ip address directly. To remove the ip address from the list of trusted domains once you're satisfied, run:

sudo nextcloud.occ config:system:delete trusted_domains 2

10. Offset your CO2

It's not going to be clear exactly what the environmental impact of your server is, but it won't be nothing. You can get a rough idea how much CO2 your server emits with tools like this one. Then you can buy carbon offsets from a reputable carbon offset vendor, like Less. I spent $10/year to offset half a tonne of CO2.

I know carbon offsetting is a long and complicated topic, and the environmental impact of computing infrastructure goes way beyond CO2 emissions, but the point is just to be aware that doing all this stuff on your computer has potentially negative consequences in the real world and to at least try to minimize them where you can and mitigate them where you can't.

Bonus

Set up a Backblaze bucket as external storage, e.g. for photos

Install and enable the "External storage support" app for your Nextcloud instance
Go to "Settings" then, under "Administration" in the left side bar (not under "Personal"), click "External storages"
Enter a name for your new folder^* and select "Amazon S3" from the "Add storage" dropdown, then fill in the details for your Backblaze bucket and account

* Make sure the name you give the external storage folder isn't already taken. I called mine "Photos", which already existed in my Nextcloud files, and it conflicted in strange and surprising ways. If you want to call your external storage folder "Photos" make sure to go delete the "Photos" folder that's already there first.

Get notified when you're approaching your storage limit

If you choose the cheapest Linode server like I did it doesn't come with much storage, and depending on how much data you have and how many backups you're leaving on the server you might run it out of storage pretty quickly. There's an app called "Quota warning" in the monitoring category you can install to get notified if you're approaching your server's storage limits. You can configure when and how it notifies you in "Additional settings" after it's installed.

That's it! I hope this helps someone avoid hours of searching through documentation, blog posts, and outdated forums. Good luck!