DEV Community: Carl Vitullo

Picking Discord role colors

Carl Vitullo — Mon, 28 Feb 2022 18:18:41 +0000

Reactiflux recently added a role menu to allow members to self-identify their experience and attitude at work, which we make visible in the server by a combination of role colors and role icons. While doing so, I had to wrestle with picking effective colors.

Discord exposes a palette of colors, but the slate of options isn't very rich. We needed to add colors for a total of 8 similar roles, so it was obvious that we'd need to go beyond what Discord makes available.

We approached this with a couple of goals in mind. We wanted to make it easy to get a rough idea of someone's experience and to give an obvious way to distinguish managers and tech leads. In picking exact colors to use, we were subject to a constraint from Discord as well: role colors must be easy to read for members who use either dark mode or light mode, since we can't select a different color for each.

For this reason, we chose to use 2 gradients. One to span our 6 individual contributor roles, and another to signify the 2 management track roles. I am in no way a visual designer, but I did my best. I found the tools on colordesigner.io very helpful for coming up with colors to use, and WebAIM useful to verify contrast ratio.

I used the random color generator to gather a couple of likely candidates for start- and end-colors for the gradient, plugging them into the WebAIM contrast accessibility checker to verify that the colors had a contrast ratio of at least 3, in both light- and dark-mode.

Once I had a pair of colors I was happy with, I plugged them into the gradient generator to create a set of intermediate colors. It took some trial and error to make sure the entire gradient was sufficiently distinct, and we may iterate on this further.

I wanted to keep colors from being too close to each other, while also ensuring they weren't too visually similar to our staff and distinguished member colors.

We're pretty happy with these! It may be a small detail, but if names are difficult to read then it's that much more difficult to get to know who's in the community, so the work is worth doing.

Thanks for reading! Reactiflux is the largest chat community of React developers, it's a great place to get help with learning React, level up your skills, or get advice for your career. Check out our past Q&As with notable React community members, and join the conversation yourself!

Optimizing a dockerfile

Carl Vitullo — Wed, 05 Aug 2020 18:09:22 +0000

There are 3 major points to consider when optimizing a Docker file:

Build context size
Cached layers
Final image size

Build context size

The build context is the entirety of the disk contents for the folder containing your Dockerfile, except parts ignored by .dockerignore. This can be massive — on a Javascript application, this means copying over node_modules, which would be installed as part of the build anyway!

On a real world example, failing to add node_modules to my .dockerignore file yields:

Sending build context to Docker daemon 788.6MB

That context transfers at about 30-50MB/s. A significant wait! Just ignoring node_modules reduces that to:

Sending build context to Docker daemon 13.75MB

Which is no wait at all. This alone is an easy win that can save minutes per build if not already configured.

Cached layers

Each step in a Dockerfile produces a "layer," which is cached across subsequent runs. The rules for when caches are invalidate are pretty simple:

The line of the Dockerfile is changed
The resources referenced by ADD or COPY change
Any line above the current one has been invalidated

This means that we can dramatically reduce the amount of time it takes to re-build a docker image by approaching our Dockerfile with some care.

Order commands from least- to most-frequently-changed

We can maximize the amount of layers that remain cached by placing frequently changed resources or instructions later in the file. This is constrained, though, by the process having a necessary order of operations. You can't install your dependencies before copying a manifest, and you can't build your source before installing your dependencies.

Un-bundle unrelated instructions

Instead of configuring the OS, installing the dependencies, and building the application in a single step, separate these into discrete steps. Pulling from a real Dockerfile I optimized,

FROM ubuntu:16.04
WORKDIR /app/src
COPY . /app/src
RUN apt-get update && apt-get install -y curl git make … && apt-get clean
RUN yarn install && yarn build
# …

Every time a file anywhere in your project changes, this will start from square 1. COPY . /app/src is telling it to do exactly that.

FROM ubuntu:16.04
WORKDIR /app/src
RUN apt-get update && apt-get install -y curl git make … && apt-get clean
COPY package.json yarn.lock /app/src/
RUN yarn install
COPY .babelrc webpack.config.js /app/src/
COPY src /app/src/src
RUN yarn build

This again can cut minutes per build off the total time spent. Breaking down the steps:

Install OS-level dependencies. These are specified inline, so it will never invalidate unless I edit this line of the Dockerfile.
Copy my package information and lockfile
Install dependencies. This step will only invalidate if one of the specified files changes.
Copy source code and build configuration
Build the app. This invalidates on every code change.

COPY has an awkward experience for this. Files may be copied many at once with the final item in the list being interpreted as the in-image destination folder, but source folders must be specified 1 per COPY command. Folders are also specified not by the containing folder, but by the exact folder itself (note COPY src /app/src/src, not COPY src /app/src). This leads to some extra layers in complex directory structures, but the advantages of this are huge.

Consider how long each of these steps takes. I find apt-get and yarn install frequently take anywhere from 30 seconds to multiple minutes, each, and both are completely irrelevant if I'm only making a change to source code.

Final image size

The final docker image is, presumably, the artifact that is being produced and passed throughout the rest of the deployment. These may be archived for accountability purposes, uploaded across a network, or otherwise be used in situations where keeping them small is advantageous. Some context first:

The real-world Dockerfile I've been referencing is a single-page application, and thus is served by nginx. A straightforward way to configure a docker image for this would be to start FROM nginx, install node, yarn, and all the other tools needed to build it, then copy the resulting files to nginx's content root.

However this leads to an inefficiency: node, yarn, even my node_modules, aren't used by nginx. It's compiled into static html, js, and css. The rest of those tools and resources are now wasted space. Worse, it's a potential security risk! The running server, rather than only having exactly what it needs to execute the production environment, has multiple tools for installing and running external code. (I'll admit to not knowing a precise exploit — security isn't my specialization — but excluding unnecessary tools reduces the attack surface)

Docker lets us address that with a feature known as "multi-stage builds." We can name intermediate images used to run our build, then extract resources for consumption in a later (more narrowly scoped) image.

FROM ubuntu:16.04
WORKDIR /app/src
RUN apt-get update && apt-get install -y curl git make … && apt-get clean
COPY package.json yarn.lock /app/src/
RUN yarn install
COPY .babelrc webpack.config.js /app/src/
COPY src /app/src/src
RUN yarn build
# I’m cheating a bit. These two images aren’t _identical_ because I
# don't have a working example of nginx handy. The installation and
# execution of nginx is omitted here.

The above dockerfile produces an image that's 2.21GB. I can make this a multi-stage build by naming our "build" image and using a second FROM.

FROM ubuntu:16.04 as build
WORKDIR /app/src
RUN apt-get update && apt-get install -y curl git make … && apt-get clean
COPY package.json yarn.lock /app/src/
RUN yarn install
COPY .babelrc webpack.config.js /app/src/
COPY src /app/src/src
RUN yarn build
# Our final image source
FROM nginx:1.17
COPY --from=build /app/src/dist/ /usr/share/nginx/html/
COPY nginx_default.conf /etc/nginx/conf.d/default.conf

The multi-stage build version produces an image that's 166MB. That's less than one tenth the size! There's surely some additional fat to trim there, as the nginx base images are about 50MB and my built static files are only about 40MB, but this is quite an improvement over the simplest configuration without spending too much time.

I've repeated this process on a number applications I've maintained, and it's paid dividends in time saved every time. Time spent waiting for a build is waste time. It breaks me out of my flow, I get distracted and don't notice when it finally finishes, and it makes it so slow to test changes that I don't want to experiment. Optimizations such as these are substantial wins to my productivity.

Securing a large Gatsby site with a CSP

Carl Vitullo — Tue, 21 Jan 2020 18:55:47 +0000

(Photo by Piotr Hamryszczak)

Recently I shipped a large project using some of the hot new tools in the React community—Gatsby and MDX—serving a global audience with about 160 different pages (blog posts, landing pages, and indexes). Because I work in the cryptocurrency space, which has a relatively large population of suspicious actors, I wanted to do everything I could for security. One of the ways I did this was to ensure I had a restrictive Content Security Policy (CSP).

Content security policies allow you to whitelist where resources can be loaded from, what external connections can be made, and what types of code can be run. It's a way to lock down the wild-west execution environments that are the browsers of the general public by removing certain attack vectors. I learned that CSPs are difficult to get right, but new policies can be rolled out safely.

CSP rules are granular to the extreme, and the rules for writing them are tricky. Because of how various browsers and common tools work, I discovered that there were many violations I didn't expect, that are difficult to verify, and that there might be new errors at any time. Because CSPs are a whitelist, I had to learn a lot about the details in order to get it working properly.

It took a lot of trial and error.

Limits from Gatsby and MDX

One problem I had very quickly was that Gatsby and MDX, at time of writing, require some large carve-outs in order to operate. There are 2 rules that, left out of the whitelist, close a lot of the methods of XSS attack.

'unsafe-eval'
'unsafe-inline'

These 2 methods are the most common ways to inject malicious JS onto pages. 'unsafe-eval' prevents strings from being executed as code, and 'unsafe-inline' requires that all scripts be loaded from files over the network. With these two omitted and a list of acceptable domains scripts can come from, this gives you a high degree of confidence that you aren't vulnerable to XSS attacks—your server would have to be compromised before malicious scripts can be executed.

However, Gatsby itself places many inline styles and scripts on the page, and MDX uses new Function. Because they make use of this functionality, we have to punch pretty big hole in the security policy. On the one hand, this is pretty great: there are very few problems, it's really really close to letting us be extremely restrictive with our security rules. On the other hand, there are so few problems: so close, yet so far. I can't lock the policy all the way down.

Gatsby has an umbrella issue for these problems, and there's a workaround for MDX (with some restrictions on how you can write your code). There's also gatsby-plugin-csp, which will determine hashes for inlined scripts so they can be whitelisted without unsafe-inline, but it outputs to a <meta> tag thus disallowing report-uri. Ultimately these options didn't work for my codebase.

Launching the CSP

After getting a CSP that allowed all our images, any iframes, data sources, embedded media, and form submission targets, I added it in report-only mode and set it loose. This is what makes it so easy to roll out a new CSP: you can put it in place where it will emit errors, but not block content. This lets you spot problems before they surface for your visitors and fix errors before they see them.

In rolled the browser errors. Some are frequent enough to make me think it's every user of a certain device or browser, teaching me something about how varied the web is. One error that seems obvious in hindsight is translation services. I found Google Translate specifically difficult to test, though. The website allows you to browse via an iframe, which runs into completely different CSP rules as the in-Chrome "do you want to translate this page?" prompt, and it took a trick to make that prompt appear.

Quickly finding mistakes

Very fortuitously, one of my good friends published a CSP evaluation tool, csper.io, which helped me catch a number of silly mistakes that accidentally weakened my policy. CSPs have an fiddly syntax, and because they only warn when the rule is violated, broken rules can be invisible.

-base-uri none;
+base-uri 'none';

In CSPs, values without strings are interpreted as URLs. Here, none is actually being interpreted as https://none. This isn't a large problem, because that's not a domain that can be registered and thus not a significant vector, but it's annoying to accidentally leave something subtly broken.

The final policy

The final CSP is quite long, much longer than I expected going into it. (I've anonymized this to remove specific servers, replacing them with generic addresses)

block-all-mixed-content;
base-uri 'none';
default-src 'self';
object-src 'none';
frame-ancestors 'self';

form-action https://company-name.us9.list-manage.com;

img-src data: https: android-webview-video-poster: android-webview:;

font-src data: 'self' https://cdn.embedly.com https://fonts.gstatic.com;

style-src 'unsafe-inline' 'self' https://translate.googleapis.com https://cdn.embedly.com;

frame-src https://runkit.com https://third-party-iframe.example.com https://www.youtube.com https://cdn.embedly.com;

connect-src 'self' https://company-api.example.com https://sentry.io https://third-party-api.example.com https://api-cdn.embed.ly https://translate.googleapis.com https://www.google-analytics.com;

script-src 'self' 'unsafe-eval' 'unsafe-inline' 'data' https://embed.runkit.com https://www.google-analytics.com https://company-name.us9.list-manage.com https://translate.googleapis.com https://translate.google.com https://api.microsofttranslator.com cdn.embedly.com;

report-uri https://sentry.io/api/id/security/?sentry_key=key;

In summary

Because this is a whitelist, I know that I'll have to keep an eye on the violations to see if useful services become blocked in the future. If a translation service starts using a new domain, it won't work until I add it to the whitelist. There are enough real violations to make this a large set to sort through, making that a bit of a challenge.

Csper helped me tighten up my rules by giving me prioritized suggestions, which I found reasonable and easy to change. They were also descriptive enough that I learned a lot about the possible attack vectors that websites face. For a fee, it can also be used for long-term monitoring, which could prove more useful than Sentry's reports due to the specialization.

Overall, I felt adding a CSP was worth the effort invested in adding it, though the process was slower moving than I'd hoped. Because errors frequently came from unusual combinations of browsers, extensions, and third-party tools, I needed to gather several days worth of reports before I felt confident after making any changes to the policy. If I had started from csper.io, I think I would have saved myself a lot of time when I was learning how CSPs work.

What was your worst new job experience?

Carl Vitullo — Fri, 09 Aug 2019 15:46:00 +0000

Sometimes you start a new job and quickly realize it's not a good fit. What's your worst new job story?

Why do you love your job?

Carl Vitullo — Fri, 02 Aug 2019 19:05:17 +0000

Jobs aren't all frustration and misery. What does you job do for you that makes you happy to stay where you are? The benefits, a great perk, your coworkers, the projects?

What made you quit your job?

Carl Vitullo — Fri, 26 Jul 2019 17:36:46 +0000

Our industry has a reputation for short employment tenures. What circumstances tipped you over the edge into quitting your job?

How a walkout happens in tech

Carl Vitullo — Tue, 09 Jul 2019 16:44:39 +0000

At 1:30pm on Wednesday, June 26th, hundreds of Wayfair employees walked out of the office after failing to reach an agreement with Wayfair's executives about furnishing an ICE detention center.

What I wanted to know is, how does a walkout happen? What is the sequence events that begins with somebody sharing an opinion and ends with hundreds of people risking their livelihood?

Largely pieced together from an interview done with Vice, I've put together a rough timeline. I'm filling in gaps to the best of my ability, but I haven't been able to make direct contact with anyone involved in organizing the walkout.

The sequence of events

Early 2019

A worker at Wayfair sees a sale to BCFS, a contractor for ICE with a reputation for sub-standard facilities, and an unknown number of people raise concerns to management. No action was taken.

June 19th

A second sale to BCFS is spotted and disseminated, reportedly to a group of around 24 coworkers throughout the company. This group meets the next day to draft a letter to management with 2 specific requests.

June 21st

A digital copy of the letter is circulated internally, with a link to an external form used to digitally sign it. After about 4 hours, the organizers deliver it to management with 547 signatures.

June 24th

Management responds, negatively. They "appreciate your passion and commitment," and are "proud to have such an engaged team," but decline to actually do anything in response.

June 25th

Tuesday morning, the management response was tweeted. Later that day, it became a national issue with AOC and Bernie Sanders tweeting in support of it. After news of the walkout breaks, management shifts their position to donating $100k to the Red Cross—not to RAICES, an immigrant rights group, as proposed by the organizers.

June 26th

Workers walk out.

wayfairwalkout

@wayfairwalkout

It was awesome to see the turnout today. Thank you to everyone who showed up to the #WayfairWalkout

22:44 PM - 26 Jun 2019

410 2725

It's hard to get a count, and it's hard to know how many in the crowd are Wayfair employeers vs friends of the walkout showing solidarity, but this looks like a big turnout to me. Direct actions like this are risky, so Tech Workers Coalition set up a hotline in case any workers face retaliation.

Analyzing it

I'd bet that the genesis of the walkout could be traced back to shortly after the first complaints were dismissed. All the ingredients were there; like-minded folks throughout the company came together with a shared complaint, and management took no action in response. Organized actions begin with a few people talking, and if they didn't already know each other, that's likely when they started collaborating.

Two questions I have for the organizers about the early stages:

Did you already know a group of coworkers who were opposed to this type of sale? How did you reach out to them about this sale?
How did you know to look for BCFS?

The letter to management is so well composed and thoroughly cited that I have to assume it wasn't written from scratch in 1 day. That, more than anything, makes me think that significant effort was put into planning this walkout in the months between sales to BCFS.

I feel this letter, as the primary contact between a large group of employees and the company's owners, is one of the most important tools. Later on, I break down the letter and offer my analysis of what they've written. I have a tiny bit of experience here—I helped draft a letter attempting to convince Palantir employees to resist ICE contract renewal from the inside.

Almost certainly, the meeting on the 20th was where the walkout was concretely decided upon—a contingency in case an agreement couldn't be reached.

Three questions I have about their first meeting and writing the letter:

How much planning happened in advance of this meeting?
What were your guiding principles when writing the letter?
When in the course of this did you plan the walkout?

Tearing apart the letter to management

The full letter these workers submitted to management was posted on Twitter. This is a great artifact of a successful action. I'd like to break down how it was written, and why, to learn how to write something similar.

The most important thing when writing, I've found, is to have a clear idea of who I'm speaking to and what I want them to take away from it. The audience here is obvious: management, specific individuals that the authors probably know, at least by reputation. What should they take away from it? From their previous behavior, it's clear that they don't agree that it's wrong to accept money from BCFS. It's hard to change somebody's mind, especially when they have a financial interest in staying the course. Let's look at each paragraph and guess at how Wayfair's leadership might receive it.

Paragraph 1

The opening paragraph is straight and to the point without using (excessively) inflammatory language. It clearly communicates the emotions involved without letting them bleed into the words used. It offers citations for its strongest claims—some might dispute the use of the term "atrocities," and it preempts them by offering examples. This introduction sets the stage for the rest of the letter, explaining for whom it speaks and what issues they're speaking about.

However, I think some parts of this might be poorly received by a leader within Wayfair. If I were asked to edit this, I might have changed it to,

We, the 547 undersigned, are writing to you from a place of concern and anger about Wayfair's role in the atrocities being committed at our Southern border. The United States government and its contractors are responsible for the detention and mistreatment of hundreds of thousands of migrants seeking asylum in our country ~~-- we want that to end~~. We ~~also~~ want to be sure that Wayfair has no part in enabling, supporting, or profiting from this practice.

Immediately sharing the number of people that have signed the letter helps sell the the weight rest of its contents, assuming it's a high number—in this case, 547 seems high to me. I'd also emphasize the focus on what Wayfair controls—Wayfair has no leverage to end the atrocities, so it distracts from the main request.

Paragraph 2

Emphasizing that this isn't the first occurrence ups the ante significantly. The footnote explains that BCFS's first purchase of Wayfair's products were used at a camp that was later found to be substandard. This increases the veracity of the claims: BCFS isn't assumed to be a bad actor by association with detention, there is evidence of their own bad behavior.

However, I feel the lede has been buried in a footnote. Comparatively, the UN statement on child rights violations feels abstract; a statement by an international governing body condemning child detention is much less immediate than information that their customer has previously put children at risk.

Paragraph 3

Solidly bringing it back to Wayfair's own ideals, a VP of B2B sales is quoted directly. Using an executive's own words in support of their cause is powerful—Wayfair can't dispute the message, it's from one of their own. This paragraph is short, punchy, and neatly measures their complaints against the business' aspirations.

Paragraph 4

Finally, the asks. They're limited in scope and accomplishable. It's a combination of rectifying a wrong and making an organizational change to prevent the wrong from reoccurring. They're directly related to a recent event within the company, and were presented in a timely manner.

Management's response

This letter is fantastically written, and the asks are reasonable. And yet:

We believe all of our stakeholders, employees, customers, investors, and suppliers included, are best served by our commitment to fulfill all orders. Management's response, delivered the next day, is a flat no. (Mirror, in case the tweet is removed)

Thus the walkout. It offers a good reminder: It's not enough to complain if your company is doing something that you and your colleagues feel is wrong. If complaints aren't met with change, it must escalate to action.

The future of Bitcoin and Ethereum

Carl Vitullo — Sat, 09 Feb 2019 02:35:55 +0000

I’ve loosely followed cryptocurrency for almost a decade. In 2011, I briefly mined Bitcoin (and Litecoin, when it became clear that GPU mining BTC was unlikely to net me much of a reward). Bitcoin’s proof-of-work algorithm to create a trustless ledger was a true innovation in decentralized systems, and Ethereum’s expansion of it into a general compute platform was another giant leap. But looking at the state of the world right now, neither seems poised for the level of success that is evangelized.

I am bullish on cryptocurrency in general. Our existing monetary system already operates on a digital ledger, and reducing institutional trust is a good thing. Moving money and tracking ownership is too important to leave in the custody of private institutions that have a long history of looking out for number one. Yet both Bitcoin and Ethereum strike me as being an imperfect fit for mainstream financial products. Some problems I see:

Block discovery is too long to be used for payments.
High volatility limits use as a method of payment.
Smart contracts are too unreliable for institutions.

These problems are substantial hurdles that Bitcoin and Ethereum will have to face in the coming years.

I'm going to assume the reader is familiar with how cryptocurrencies operate, at least at a high level. The YouTube channel 3Brown1Blue has a fantastic explanation of the proof-of-work algorithm, and Coinbase has decent simple explanations of common cryptocurrencies and related topics.

Long block discovery

Bitcoin has notoriously long transaction times, taking up to an hour to fully settle on the network. This isn’t an inherent property of Bitcoin. The core developers have chosen to keep block discovery times averaging about 10 minutes and the amount of data per block low, which limit transaction latency and throughput respectively. Because proof-of-work requires several blocks to be found before a transaction settles, the transaction time is a multiple of Bitcoin’s 10-minute block time. Ethereum’s block time is 15 seconds, for a point of comparison.

The transaction time is a multiple of Bitcoin’s 10-minute block time.

This puts Bitcoin transactions at a speed somewhere between an ACH transfer and a credit card. It’s not fast enough to use at the corner store, but it could work for some sort of payments app; a slow Venmo or Square Cash. Ethereum is much better, but its 15 second block time means transactions usually settle in about 3 minutes. This is suitable (if a little slow) for a payments app, but far too long a wait if you’re checking out with a line.

There are serious efforts to address this. The lightning network is way for two parties to confidently settle numerous transactions off-chain, only bringing it back onto the chain once they've completed their transactions with each other. With enough individuals maintaining these payment channels, a payment could occur between 2 parties without a direct connection by making multiple "hops" where connections exist. The lightning network is currently operating in a limited capacity, and in the future it could extend to cross-chain payments—acting as a compatibility layer between different protocols.

Price volatility and new currencies

Volatility causes fewer direct problems than slow transactions, but it makes it difficult to use as a means of transfer. The long transaction times and volatility exacerbate market risk, the potential that the currency’s value will decrease by the time the recipient is able to sell what they’ve been given. Steam stopped accepting it in late 2017 for exactly this reason.

Those paying suffer the same market risk in the other direction. The currency might increase in value, encouraging you to hold onto it rather than spend it. If you were paid with Bitcoin last week, you don’t know how much purchasing power it will hold next week. The macro-scale impact of this volatility is visible in the decline of BTC as a payment option and the rise of the “hodlers.”

If you were paid with Bitcoin last week, you don’t know how much > purchasing power it will hold next week.

This price volatility has some organizations attempting to hybridize fiat currency with cryptocurrency, leading to a rise in “stablecoins”—digital assets anchored to real-world currencies. A notable example of this is Tether, USDT, a representation of the US dollar on the blockchain. These introduce new risks: Tether has had major delays fulfilling withdrawal requests, and is accused of not having adequate balances to fulfill its obligations.

The driving force behind the volatility is that cryptocurrencies are treated as stock in the projects that issue them. Cryptocurrency enthusiasts behave like speculators, hoping to buy low and sell high. The price fluctuates significantly based on project news and the cryptocurrency market overall, which has the Securities and Exchange Commission keeping a close eye on the space.

Smart contracts aren’t reliable

Bitcoin doesn’t have a concept of “smart contracts” in its protocol, but it’s a core innovation of Ethereum. Ethereum provides guarantees about execution and integrity of data for smart contracts, but they’re still arbitrary computer programs. While this opens up the fascinating world of decentralized apps, it’s not ideal for applications that require a high level of security. It's bad if CryptoKitties has a bug allowing attackers to steal your unique cat, but it’s life altering if attackers steal all of your money.

Many of the potential applications of smart contracts demand such high security. Vitalik has tweeted about non-financial applications, but verifying college degrees would be similarly catastrophic if something were to go wrong.

As arbitrary computer programs, Ethereum’s smart contracts provide no guarantees that they will execute correctly. This relates to the idea of “correctness” in computer science: How do you prove, without executing the code, that it will do what you expect? In the world of Ethereum, your only option is a careful review of the code, with some limited assistance from auditing software. This is hardly compelling to major institutions that already operate systems with a high level of confidence.

Ethereum’s smart contracts provide no guarantees that they will > execute correctly.

Major bugs in high-profile projects shows how difficult smart contracts are to get right. Even Coinbase, arguably the most trusted name in cryptocurrency, had a bug allowing users to credit their account with unlimited Ethereum. The most infamous is the DAO hack, which was reverted by a hard fork of the Ethereum chain and lead to the Ethereum/Ethereum Classic split. Parity, a wallet written as a smart contract, is described as having been “audited by the Ethereum Foundation’s DEV team, Parity and others from the community.” Yet it suffered multiple hacks and bugs.

Because the Ethereum protocol acts as a runtime for smart contracts, even smart contracts that are well implemented are vulnerable to bugs introduced at a protocol level. A recent protocol upgrade, codenamed Constantinople, was delayed after ChainSecurity discovered it would make previously secure smart contracts vulnerable to a “reentrancy attack.” It’s difficult for me to imagine any such institution making a significant investment into a technology that has a consistent track record of security vulnerabilities and hacks.

What does this mean for Bitcoin and Ethereum?

Bitcoin’s flaws are not inherent to the protocol, but politics within its development means that it’s unlikely to change. The entrenched community of miners and supporters may make it impossible to include the large changes needed to make it competitive with more modern blockchain technologies.

Ethereum’s problems are perhaps more fundamental and might be harder to fix. The premise of the technology is that anyone can write code for a global computer, but I struggle to see use cases that aren’t better served by a centralized system. This has borne out in the projects that have been built on Ethereum; the most widely successful smart contracts (excluding ERC20 tokens) are for trading or games. These products are popular with cryptocurrency enthusiasts, but only serve those who are already actively interested in the space.

Cryptocurrencies have become self-serving, insular communities. Far from what was proselytized—a global equalizer, a means of transfer unrestricted by any one party—they have become just another type of investment for those with wealth to invest in. Neither Bitcoin or Ethereum have demonstrated that they’re well-suited to solve any of the world’s problems. I don’t believe that either of them fill a real need in the world in their current state, and their ecosystems and communities work now only to fill their own needs.

Symptoms of a dysfunctional team

Carl Vitullo — Mon, 21 Jan 2019 16:36:58 +0000

(image credit: rawpixel on unsplash)

I've had a lot of jobs (8 since 2012), which means I've worked with a lot of teams. One of the main reasons I changed jobs so frequently for a couple years is that many of the teams I joined felt very dysfunctional—it was difficult to get work done and there wasn't a clear sense of purpose.

Hopefully, my cheat sheet of questions for your interviewer will help you identify dysfunctional environments during the interview process, but when you're early in your career it's hard to tell whether your team is functioning as it should. There are some symptoms that I've found to indicate that a team has major cultural or technical issues that cause it to be dysfunctional.

Overtime assumed

The biggest indicator is when it becomes assumed that you'll regularly work beyond the 40 hours a week for which you're paid. Software development positions are notorious for a culture of overwork and burnout. In most professions, the federal Fair Labor Standards Act requires that overtime is paid at 1.5x your hourly rate, with salaried positions considered "exempt." However, the FLSA specifically exempts all computer professionals from overtime pay, which laid the groundwork for burnout culture to flourish.

Overtime means that too much work has been planned for too little time. It's generally understood that overtime has a detrimental effect on productivity. A synthesis of studies from 2004 evaluating productivity in construction projects has clear figures showing the drop in productivity. There are potential benefits to brief "crunches" of 1-2 weeks of 50 hours of work, but beyond that and the drop in productivity outweighs the additional time worked. That's right: studies show that those working 60+ hour weeks accomplish less than people working 40.

Long feedback cycles

One of the biggest drains of productivity is long feedback cycles. When you change a line of code, how long do you have to wait before you can tell if it worked? With modern tools and languages, cycle time should be measured in tens of seconds, if not less, but what you can expect depends on the domain you work in. Reducing small amounts of wasted time in repetitive tasks adds up, especially when they're done frequently. As always, there's a relevant XKCD:

If it takes multiple minutes to run a development build, consider how many hours your team spends collectively sitting idle. Outside of build times, some of this optimization is up to you as a developer; is there a way for you to short-circuit directly to the code path you're testing? If it takes you 10 seconds to click through UI or input data, that can quickly add up to hours of burned time over weeks of development. Modifying the surrounding code to remove these manual steps can have a significant impact on how quickly you can work.

A major contributing factor to long build times is how familiar your team is with the tools they use.

Lack of expertise with tools used

If a tool has been misconfigured, is being used without optimization, or solves a different problem than what your team has, the length of the feedback cycle is often the most visible indicator that something is wrong. Build times (both in development and when deploying to production) and the time needed to configure a local development environment are excellent proxies for your team's overall level of expertise with their tools as well.

One of the most common reasons behind a team lacking expertise with their tools is when they've followed trends, rather than addressed the problems they encounter. Frontend and Javascript developers are often criticized for this behavior, but I've seen it in backend and ops as well—big data, serverless, AI, Docker, microservices. These are useful, innovative new tools, but using cutting-edge tools that you don't understand is worse than simpler infrastructure you're more familiar with.

It's hard to judge whether the team understands a tool if you yourself don't, but you can clarify what problems a tool is meant to solve by talking with others. Within your team, there should be a clear answer for why the tool is used and why it was picked over alternatives. Talking with others outside the team, like at a conference, meetup, or online, you may find that you don't have the problems that a tool is designed to solve.

Poorly configured tools also contribute to another major symptom: outages.

Frequent outages

Outages should be a rare event, happening only after an external incident or a confluence of factors. In a healthy environment, outages are followed by postmortem that ends with process or infrastructure changes to prevent the same incident from recurring. A prerequisite of an effective postmortem, however, is a deep understanding of what went wrong and what changes are possible. Shallow knowledge of your tools limits both of these. If an outage was resolved by turning it off and back on again, the opportunity to learn from it is usually lost.

Outages generally come in a few different flavors. They could be caused by performance problems, failed deployments, or If your team has a long cycle time, it becomes prohibitively time-consuming to polish off the small bugs at the end of a unit of work. These bugs accumulate, eventually leading to a crash in production or an invalid deployment. Worse yet, these same issues impair your ability to get the service up and running again. It's a cycle that feeds itself, and teams or companies that are not able to address the fundamental issues may end up spiraling to their death.

Excessively rigid process

Especially as you gain expertise and advance in your career, you need to have a certain degree of flexibility in completing your work. Prioritization will never be perfect and you'll always find issues in the code you touch. It's rare that invisible concerns like performance, maintainability, and supporting infrastructure bubble to the top of a product manager's priorities. Well-operated organizations will typically have engineering leadership to balance this tension, but often it falls to you, an individual developer.

A restrictive process may be a knee-jerk response put in place after features were repeatedly delayed (often due to long cycle times) or frequent outages. If developers don't balance features and internal work, the process may be a way to force features to be prioritized. The process may also be a band-aid over fragile build tools as well—"just follow the process to the T and it'll all be okay."

It's also a symptom of problems within the leadership. Rigid process and micromanagement might be due to inexperienced managers, or those self-conscious about their level of contribution.

Leadership without obvious contributions

It should be apparent what role the leaders of your team play when you look for it. A project manager should facilitate the creation and prioritization of tasks, making sure they contain the information you need to complete it. A tech lead should ensure that the process and tools being used enable you to ship code quickly and without error. A product manager should verify that what's being built meets the end-users' needs. It may not be obvious from where you sit, depending on your role within the team, but each of these individuals should be furthering the team's goals in their own way.

If, when talking to coworkers and those above you in the organization, nobody can quite articulate what it is the people leading a team are contributing, the team has a major leadership problem. When those in charge of decisions abdicate their responsibilities, the process and technology begin to break down. Over time, the project may become mired in tech debt, waste people's time as they track down information, or ship features that nobody uses.

Often, these types of leaders are struggling within their defined role. Unfortunately, a relatively common form of this manifests as bad success metrics for the team as a whole.

Organizational metrics unrelated to outcomes

Objects and Key Results (OKRs) and Key Performance Indicators (KPIs) are two buzzwords that became popular after the publication of books of the same name. The ideas behind them are excellent: organizations should measure that their efforts are working. However, if these OKRs or KPIs don't relate to positive outcomes for the team or organization, then they're simply irrelevant numbers being used to micromanage developers.

Unfortunately, this is often out of your control as an individual contributor. It may even be out of your purview, and they may be locked in for the quarter or even for the year. Misalignment between key metrics and actual, useful software is a major problem for the organization.

Responding to these problems

These problems happen at companies of all sizes and are often deeply rooted in the culture that a team has grown around. There aren't one-size-fits-all solutions to these problems, and recognizing dysfunction isn't enough to fix it. Ultimately, many of the solutions to these problems require either buy-in or changes from management, which can be hard to effect when you're a few rungs down from them.

One important thing to keep in mind is that individual contributors are usually first to feel the effects of these breakdowns. At an otherwise healthy company, those in charge may know that something is wrong, but not have the information they need to diagnose the problem. If any of these resonate with you, they would make good conversations to have during a one-on-one with your manager.

If an organization is unable or unwilling to change, it may be a sign to freshen up your resume and begin exploring your options. It's easy to mistake the problems of one team with problems that are organization-wide, though. If you encounter problems like these, talk with coworkers on other teams about their experience. They may be able to help you resolve them, or you might be able to move within the company.

Thanks for reading! I'm on Twitter as @cvitullo (but most other places I'm vcarl). I moderate Reactiflux, a chatroom for React developers (check out #jobs-advice!) and Nodeiflux, a chatroom for Node.JS developers. If you have any questions or suggestions, reach out!

Everything you need to know about React Hooks

Carl Vitullo — Thu, 25 Oct 2018 16:57:15 +0000

React just announced a new feature: Hooks. It's a brand new set of APIs that enables powerful new ways to share stateful logic between components, optimize performance without significant rewrites, get some of the benefits of Redux-style separation of concerns, and more. They also deliver on a promise that the React team made years ago—stateful function components. Using state from function components came up as a possibility in Dan Abramov's Q&A on Reactiflux all the way back in April 2016.

It's been a long time coming, but they're here! More than just state, though, there are 11 new functions in all that should enable the full range of functionality that we use classes and the lifecycle for today.

useState
useEffect
useContext
useCallback
useMemo
React.memo (Not a hook, but new)
useReducer
useRef
useLayoutEffect
useImperativeMethods
useMutationEffect

Let's take a look at what each of them is for.

`useState`

Stateful function components are enabled with the new function useState.

import { useState } from "react";

const SomeComponent = props => {
  const [state, setState] = useState(initialState);
  return (
    <div>
      {state}
      <input onChange={e => setState(e.target.value)} />
    </div>
  );
};

If you ever used the library recompose, this API may look familiar. useState takes an initial state as an argument, and returns the current state and an updater function. The setState it returns is almost the same used by class components—it can accept a callback that gets the current state as an argument, but it doesn't automatically merge top-level object keys.

Each call to useState is paired with a component, with its state persisting across renders. This means that you can call useState multiple times within a single function component to get multiple independent state values. Because the setState returned isn't scoped to a single component, we can define stateful behaviors independent of the component. This enables powerful new ways to abstract stateful logic.

Let's look at an example that I've run into on several projects: managing sort state in several components. I find the APIs that table components expose to be inflexible, so I tend to write tables of data as one-offs. My current project has some code for managing what key to sort against, and in which direction, copy-pasted into several different components. With useState, we gain the ability to define it as a separate module.

const useSort = (someArray, initialSortKey) => {
  const [state, setState] = useState({
    isAscending: false,
    sortKey: initialSortKey
  });

  // Let's pretend `makeSortComparator` exists for simplicity
  const comparator = makeSortComparator(state);
  const sortedData = someArray.slice().sort(comparator);

  return {
    ...state,
    sortedData,
    toggleAscending: () =>
      setState(state => ({
        ...state,
        isAscending: !state.isAscending
      })),
    setSortKey: sortKey =>
      setState(state => ({ ...state, sortKey }))
  };
};

Now we have a reusable method to use in our data table components. We have a simple API we can use across our many different tables, with each component working off its own separate state.

const SomeTable = ({ data }) => {
  const { sortedData, ...sortControls } = useSort(
    data,
    "id"
  );
  return (
    <table>
      <TableHeading {...sortControls} />
      <tbody>
        {sortedData.map(datum => <TableRow {...datum} />)}
      </tbody>
    </table>
  );
};

Please note: the React team strongly recommends prefixing the names of these types of modules with use so there's a strong signal of what kind of behavior it provides. See the full docs for more about writing your own hooks.

I am super excited about this new way to share functionality. It's much more lightweight than a HOC in all ways; less code to write, fewer components to mount, and fewer caveats. Check out the API documentation for all the details.

`useEffect`

A lot of components have to kick off different types of effects as part of mounting or re-rendering. Fetching data, subscribing to events, and imperatively interacting with another part of the page are all common examples of this. But the code for handling these types of effects ended up scattered across componentDidMount, componentDidUpdate, and componentWillUnmount.

If you wanted to run the same effect when a prop changed, you either had to add a mess of comparisons in componentDidUpdate or set a key on the component. Using a key simplifies your code, but it scatters control of effects into another file—completely outside the component's control!

useEffect simplifies all these cases. Imperative interactions are simple functions run after each render.

const PageTemplate = ({ title, children }) => {
  useEffect(() => {
    document.title = title;
  });
  return (
    <div>
      <h1>{title}</h1>
      {children}
    </div>
  );
};

For data fetching and other interactions you don't want to happen unnecessarily, you can pass an array of values. The effect is only run when one of these changes.

const ThingWithExternalData = ({ id, sort }) => {
  const [state, setState] = useState({});
  useEffect(() => {
    axios
      .get(`/our/api/${id}?sortBy=${sort}`)
      .then(({ data }) => setState(data));
  }, [id, sort]);
  return <pre>{JSON.stringify(state, null, 2)}</pre>;
};

Subscriptions and other effects that require some kind of cleanup when the components unmount can return a function to run.

const ThingWithASubscription = () => {
  const [state, setState] = useState({});
  useEffect(() => {
    someEventSource.subscribe(data => setState(data));
    return () => {
      someEventSource.unsubscribe();
    };
  });
  return <pre>{JSON.stringify(state, null, 2)}</pre>;
};

This is so powerful. Just like with useState, they can be defined as separate modules—not only does this put all the code required for these complex effects in a single location, they can be shared across multiple components. Combined with useState, this is an elegant way to generalize logic like loading states or subscriptions across components.

`useContext`

The context API is great and was a significant improvement in usability compared to what existed before. It advanced context from a "you probably shouldn't use this" warning in the docs to an accepted part of the API. However, context can be cumbersome to use. It has to be used as a render prop, which is a pattern that doesn't compose gracefully. If you need values out of multiple different render props, you quickly end up indented to the extreme.

useContext is a substantial step forward. It accepts the value created by the existing React.createContext function (the same one you would pull .Consumer off to use as a render prop) and returns the current value from that context provider. The component will rerender whenever the context value change, just like it would for state or props.

// An exported instance of `React.createContext()`
import SomeContext from "./SomeContext";

const ThingWithContext = () => {
  const ourData = useContext(SomeContext);
  return <pre>{JSON.stringify(ourData, null, 2)}</pre>;
};

This gets rid of my final complaint with context. This API is simple and intuitive to the extreme and will be a powerful way to pipe state around an application.

More advanced hooks

The 3 hooks mentioned above are considered to be the basic hooks. It's possible to write entire applications using only useState, useEffect, and useContext--really, you could get away with just the first two. The hooks that follow offer optimizations and increasingly niche utility that you may never encounter in your applications.

`useCallback`

React has a number of optimizations that rely on props remaining the same across renders. One of the simplest ways to break this is by defining callback functions inline. That's not to say that defining functions inline will cause performance problems--in many cases, it has no impact. However, as you begin to optimize and identify what's causing frequent re-renders, you may find inline function definitions to be the cause of many of your unnecessary prop change.

In the current API, changing an inline function to something that won't change across renders can be a significant change. For function components, it means rewriting to a class (with all the changes that entails) and defining the function as a class method. useCallback provides a simple way to optimize these functions with minimal impact on your code by memoizing a function provided to it. Just like useEffect, we can tell it what values it depends on so that it doesn't change unnecessarily.

import doSomething from "./doSomething";

const FrequentlyRerenders = ({ id }) => {
  return (
    <ExpensiveComponent
      onEvent={useCallback(() => doSomething(id), [id])}
    />
  );
};

This is another exciting improvement in usability. What used to mean a significant rewrite of a component can now be accomplished in-place with a function directly from React.

`useMemo`

On the subject of optimizations, there's another hook that has me excited. Many times, I need to calculate derived data from the props I provide a component. It may be mapping an array of objects to a slightly different form, combining an array of data to a single value, or sorting or filtering. Often render is the logical place for this processing to happen, but then it will be run unnecessarily whenever other props or state change.

Enter useMemo. It's closely related to useCallback, but for optimizing data processing. It has the same API for defining what values it depends on as useEffect and useCallback.

const ExpensiveComputation = ({
  data,
  sortComparator,
  filterPredicate
}) => {
  const transformedData = useMemo(
    () => {
      return data
        .filter(filterPredicate)
        .sort(sortComparator);
    },
    [data, sortComparator, filterPredicate]
  );
  return <Table data={data} />;
};

I'm excited about this for many of the same reasons as useCallback. Previously, optimizing this type of processing typically meant extracting the logic to a separate function and memoizing that. Because it's common practices for memoization tools to rely on a functions arguments for invalidating memoization, that meant creating a pure function. This refactoring can end up being too substantial, so only the most extreme performance problems end up being addressed. This hook should help avoid the "death by a thousand cuts" type of performance problems.

`React.memo`

This isn't a hook, but it's a new API and an important optimization. Memoizing calculations and ensuring props don't change unnecessarily are good for performance, but both are more effective when combined with the shouldComponentUpdate or PureComponent features—neither of which is available for function components.

React.memo is a new function that enables behavior similar to PureComponent for functions. It compares prop values and only re-renders when they change. It doesn't compare state or context, just like PureComponent. It can accept a second argument so you can do custom comparisons against props, but there's an important difference from shouldComponentUpdate: it only receives props. Because useState doesn't provide a single state object, it can't be made available for this comparison.

`useReducer`

This hook has interesting implications for the ecosystem. The reducer/action pattern is one of the most powerful benefits of Redux. It encourages modeling UI as a state machine, with clearly defined states and transitions. One of the challenges to using Redux, however, is gluing it all together. Action creators, which components to connect(), mapStateToProps, using selectors, coordinating asynchronous behavior... There's a whole menagerie of associated code and libraries on top of Redux that can overwhelm.

useReducer, combined with the usability improvements to context, new techniques for memoizing calculations, and the hooks for running effects, allow for many of the same benefits as Redux with less conceptual overhead. I personally have never been bothered by the supposed boilerplate problem that Redux has, but considering how these hooks will combine has me excited for how features could be defined and scoped within an application.

`useRef`

Sometimes when writing components, we end up with information that we need to keep track of but don't want to re-render when it changes. The most common example of this is references to the DOM nodes we've created, for instance, an input node that we need to track the cursor position for or imperatively focus. With class components we would track assign them directly to properties on this, but function components don't have a context we can reference that way.

useRef provides a mechanism for these cases. It creates an object that exists for as long as the component is mounted, exposing the value assigned as a .current property.

Directly from the docs (and the FAQ:

// DOM node ref example
function TextInputWithFocusButton() {
  const inputEl = useRef(null);
  const onButtonClick = () => {
    // `current` points to the mounted text input element
    inputEl.current.focus();
  };
  return (
    <>
      <input ref={inputEl} type="text" />
      <button onClick={onButtonClick}>Focus the input</button>
    </>
  );
}

// An arbitrary instance property
function Timer() {
  const intervalRef = useRef();

  useEffect(() => {
    const id = setInterval(() => {
      // ...
    });
    intervalRef.current = id;
    return () => {
      clearInterval(intervalRef.current);
    };
  });


}

This code is more verbose than using instance properties is in class components, but it should be relatively infrequent that you need to store values in this way.

Rarely used hooks

The hooks mentioned above have covered all the use cases that I've encountered when writing applications. Reading through the docs of the remaining hooks, I understand why they exist and I'm sure that I'm using libraries that will implement them, but I don't anticipate using them myself in application code.

`useLayoutEffect`

If I use any of these 3, I anticipate it will be useLayoutEffect. This is the hook recommended when you need to read computed styles after the DOM has been mutated, but before the browser has painted the new layout.

Crucially, this gives you an opportunity to apply animations with the least chance of visual artifacts or browser rendering performance problems. This is the method currently used by react-flip-move, an amazing transition library when items change position, but there might be situations where I need to use this myself.

`useImperativeMethods`

To the best of my knowledge, this hook is a counterpart to forwardRef, a mechanism for libraries to pass through the ref property that would otherwise be swallowed. This is a problem for component libraries like Material UI, React Bootstrap, or CSS-in-JS tools like styled-components, but I haven't run into a case where I needed to solve this problem myself.

`useMutationEffect`

This is the hook I'm having the hardest time wrapping my head around. It's run immediately before React mutates the DOM with the results from render, but useLayoutEffect is the better choice when you have to read computed styles. The docs specify that it runs before sibling components are updated and that it should be used to perform custom DOM mutations. This is the only hook that I can't picture a use case for, but it might be useful for cases like when you want a different tool (like D3, or perhaps a canvas or WebGL renderer) to take over the actual rendering of output. Don't hold me to that though.

In conclusion

Hooks have me excited about the future of React all over again. I've been using this tool since 2014, and it has continually introduced new changes that convince me that it's the future of web development. These hooks are no different, and yet again substantially raise the bar for developer experience, enabling me to write durable code, and improve my productivity by extracting reused functionality.

I thought Suspense was the only upcoming feature that I'd be excited for in 2018, but I'm happy to be proven wrong! Combined, I expect that React applications will set a new bar for end-user experience and code stability.

Thanks for reading! I'm on Twitter as @cvitullo (but most other places I'm vcarl). I moderate Reactiflux, a chatroom for React developers and Nodeiflux, a chatroom for Node.JS developers. If you have any questions or suggestions, reach out! Cover image is from rawpixel on Unsplash

The Lerna license and ethics in software

Carl Vitullo — Mon, 03 Sep 2018 19:09:50 +0000

(Cover photo by Arto Marttinen)

Earlier this week, Jamie Kyle and Daniel Stockman merged a PR to Lerna adding a modified version of the MIT license. The modifications specified that certain companies were not granted a license to use the software due to their business contacts with ICE, (Immigration and Customs Enforcement, a US law enforcement agency). The rationale was given in the PR; ICE has been cruel to immigrants and refugees and Jamie wanted to use his position in open source to respond.

I agree with his cause, though not his actions; ICE has crossed a line into brutality and abuse that can't be defended. However, Jamie's change was quickly reverted. This is not a summary or reaction piece—his actions have already been debated and rehashed online, and I don't intend to pile on. It's made me think, though, about the ethics of creating software and our ability to shape the industry in which we spend our professional lives.

While I agree with Jamie in principle, I disagree with his execution. I think he would have accomplished more if he had built support for his plan before merging the new license. It is not enough to act alone when protesting the actions of large organizations. The odds that a single open source contributor could successfully pressure Microsoft and Palantir, a company that has raised two billion dollars, into severing their contracts related to ICE, a federal law enforcement agency, were low. The outcome does not surprise me.

It is not enough to act alone when protesting the actions of large organizations

Our individual moral compasses can't be the only thing that guides us, because we have very little leverage to enforce our beliefs. We as an industry have no shared code of ethics; nobody to whom we can report violations. No license or certification to give our statements weight, beyond our own reputation. In many ways, the low barrier to entry is what makes software development great, but it means we have limited protection against unethical behavior by our employers.

In other fields of engineering, there are professional societies or engineering unions. The National Society of Professional Engineers (NSPE) includes a code of ethics that members must abide by, backed by the organization itself. Members receive a title of "Professional Engineer," which is widely respected in the industry, and are expected to report ethical violations (by other engineers or their employers) to the organization for review.

The fundamental canons listed in the NSPE code of ethics resonated with me:

I. Fundamental Canons

Engineers, in the fulfillment of their professional duties, shall:

Hold paramount the safety, health, and welfare of the public.

Perform services only in areas of their competence.

Issue public statements only in an objective and truthful manner.

Act for each employer or client as faithful agents or trustees.

Avoid deceptive acts.

Conduct themselves honorably, responsibly, ethically, and lawfully so as to enhance the honor, reputation, and usefulness of the profession.

Violations of these ethics have cost lives or endangered the public. In 1981, a hotel walkway collapsed because design changes had been approved without proper review. The Challenger space shuttle failed because an engineer's manager overruled his recommendation to delay the launch. In 2002, a nuclear power plant nearly failed because engineers signed off on falsified inspection reports.

Software development comes with a different set of ethical issues. Most of the software we encounter in our day-to-day would not put our lives at risk if it were to fail. The ethical quandaries in software engineering are more frequently about the success of projects, not about minimizing the risk of failure. Software, and its impact on society, is still new, and we're still figuring out what is destructive.

The ethical quandaries in software engineering are more frequently about the success of projects

The professional organizations most closely related to our field are the Institute of Electrical and Electronics Engineers (IEEE) and the Association for Computing Machinery (ACM), both of which have ethical codes on their website (IEEE's and ACM's). But neither of these organizations has had a strong presence in my career thus far, and neither code of ethics has much to say on the unique capacity software has to influence people's lives. The ACM ethical code is much, much more detailed than IEEE's, but the sections on avoiding harm and protecting privacy are insufficient to provide guidance on the ethical questions in software today.

Is it ethical for Facebook to gather and sell the volume of data that they do? Facebook has made changes to the data it sells to its customers, but only after the question of government regulation was raised. Facebook has exceptionally fine-grained knowledge about its users' behavior and preferences, which it exposes to its advertising customers largely unfiltered. Is it ethical to allow advertisers such laser focus for their campaigns?

Is it ethical for YouTube to suspend channels without warning because of copyright claims? Google has infamously poor customer service, despite an increasing number of people relying on their platform for their livelihood. Does Google have an ethical responsibility to improve their process? Or is the ethical onus on content providers to be more diligent when identifying infringing videos?

It's clearly unethical for car manufacturers to cheat emissions tests—yet Volkswagen, Chrysler, Jeep, Nissan, Renault, and Mercedes were all found to be using software to do so. Skirting regulations is illegal and automotive manufacturers were penalized for their emissions cheating, but that unethical behavior is not present in other their areas of engineering. I posit that this is because of the lack of a code of ethics for software engineering.

It takes strength to take a stand for what you believe in and do something controversial, and I applaud Jamie's courage for his attempt with Lerna. But to have real, lasting impact on unethical behavior, we have to do more than take individual action. We need to lay some ground rules for what we as developers consider to be ethical software development.

We need a frank discussion of what ethical software development is, and we need a reference that we can use for guidance on whether a project is ethical. I'd love to see your thoughts on ethics in software in the comments, but I've also created a Discord server and a GitHub repository to capture discussions.

What unethical behavior have you seen in the industry?

If software development had a code of ethics, what would our fundamental canons be?

Staying on top of new Javascript features

Carl Vitullo — Mon, 27 Aug 2018 11:36:18 +0000

(Cover photo by Andrew Charney)

With modern JS tooling, we have an opportunity to use Javascript features before they are implemented in browsers or node. However, not all features are equal. Some are more likely than others to be integrated into the language. Some will undergo significant changes before being added, and some will be dropped altogether. If you're using that feature in your app, either of those can introduce big refactors. But how can you tell whether a feature is something you should use?

New features are introduced by TC39, a committee responsible for maintaining the specification underlying Javascript. There is a complete list of proposed additions to the spec in the TC39 proposals repo on GitHub. This is the canonical location for new features, the primary source. There is a well-defined process that proposals must go through, and learning more about that process will help you stay on the cutting edge.

The TC39 Process document defines what steps a proposal must take. This document is the source of the various stages that each feature goes through, which you may know best from the various stage-X presets from Babel. These stages are called the "maturity stages." The table defining the stages is a quick read that will give you a ton of context for how much has gone into a given proposal.

What does it mean to be "stage 1" though? If there's a Babel plugin for it, why shouldn't we use a feature? To answer that, let's walk through what the different stages mean.

TC39 maturity stages

Stage 0

Brand new proposals are referred to as "strawman" or "stage 0" proposals. Stage 0 has very little significance. It means that somebody has formally proposed an idea to the committee—that's it. The process document notes that this stage has no restrictions and does not require an attempt at specifying behavior. TC39 maintains a list of stage 0 proposals, some of which have not progressed towards standardization in 4 years, and several of which have been formally dropped.

Relying on stage 0 features in production apps is a gamble in the long run. So why have developers downloaded the stage 0 Babel preset 740,000 times (at time of writing) in the past week?

Are you using it? ...should you be?

Stages 1-3

These stages live in the readme of the TC39 proposals repo. These are the features actively working their way through the process.

Stages 1 and 2 require an attempt to specify the behavior of the proposed feature, but implementations are intended to be exploratory. The more complex the proposal, the more likely it is to change as it moves from stage 1 to 2 to 3. If you rely on a proposal whose behavior changes, you may be stuck with a significant rewrite if you ever want to update your build tools.

A perfect example of this churn is the proposals to add "decorators" to Javascript. They gained a lot of popularity when a proposal hit stage 2 in 2016, but it has not progressed to stage 3 (as of mid-2018). The proposal has changed substantially, and some of the original use cases are no longer supported in updated proposals.

Familiarizing yourself with the list of stage 3 proposals is the best way to keep track of what's around the corner in Javascript. A proposal advances to stage 3 once designated reviewers and ECMAScript editors have approved of the complete specification. It may already be usable in a browser and likely has a Babel transform. All implementations must adhere to the spec, and changes are unlikely.

That's not to say that stage 3 proposals are guaranteed to advance. Features at earlier stages are less likely to enter the spec because of how much work it takes to completely specify the behavior and implement it in browsers. Features at stage 3 may be rejected because irreconcilable issues are discovered while implementing, or in the case of Object.observe, because the feature was deemed unnecessary.

Stage 4

Proposals that are considered complete and ready for formal inclusion in the spec advance to stage 4. These move to the finished proposals section of the TC39 proposals repo. A large amount of work is required to advance from stage 3 to 4. There must be acceptance tests and must be implemented in 2 independent VMs.

Some history

TC39 and the release process were created relatively recently. If it feels like Javascript has changed a lot in the past few years, you're not wrong.

Ecma International and ECMAScript

Javascript is an implementation of a language specification called "ECMAScript," ECMA-262. Ecma International is a European standards body similar to ISO. ECMAScript is why the abbreviation for new versions are ES5, ES6, etc. In 2015, they changed the process to an annual release cycle, coinciding with the release of ES6–this is why it's also called ES2015. It's worth noting the gaps between new versions of ECMAScript. (I'd also like to note that those are the correct capitalizations; Ecma de-acronym-ized their name in 1994, but ECMAScript is still styled that way in the spec)

1999, ES3
2009, ES5, originally ES3.1
2015, ES2015, with annual releases ever since.

ES6, ES2015, or ES Harmony

ES2015 was a turning point for Javascript, both technically and with regards to future extensions. It introduced a laundry list of new features, enumerated in the old Babel docs, and was the first release under the new process.

These features completely reinvented Javascript. The reasons behind the sudden explosion of new features are largely political, and I won't attempt to recap them in depth. The short version is that the evolution of ES5 was contentious. It was originally intended to be released as ES3.1, but the final version included some features from the unsuccessful ES4 revision. If you're interested, the ES4 draft is good to skim through.

Following ES5, ES2015 was the first release under a newly developed process, led by TC39 (Technical Committee 39 of Ecma). It was the first release after the various factions reconciled and began working together again, hence the codename "Harmony."

Because this new process has been working so smoothly, recent releases of ECMAScript have not been the significant overhauls that ES2015 were. Dr. Axel Rauschmayer's blog has excellent summaries of the new features introduced in each 2016 2017, and 2018.

Modern Javascript

These official releases have become less important. Babel enables us to use features months or years before they're officially released, and we may not use the native implementations for years afterward due to legacy browsers. Babel has moved to make it even easier for us to forget about this process with babel-preset-env. By configuring it with what browsers you need to support, it will provide minimal transpilation and polyfills to use native implementations whenever possible.

Babel has also moved to deprecate the stage-x presets for reasons given in a blog post announcing the change. babel-preset-env provides a much more developer-centric solution to the same problem, and is recommended instead of manually setting presets.