DEV Community: Neil Chaudhuri

Why Coding Interviews are the Worst

Neil Chaudhuri — Tue, 09 Aug 2022 14:51:02 +0000

Tech interviews are broken. Much like with vegan cheese, we all agree there is a problem, but we all differ on how to fix it. How did we get here?

Over the last several decades as tech became a dominant industry, the new personalities that became very rich very fast also became very Gavin Belson from Silicon Valley—supremely convinced of their own outside-the-box cleverness not only at producing revolutionary technology but also at revolutionizing the way it's built. The media decided this was the beginning of a new era where the novel approaches tech companies took to everything should be a model for everyone else. Remember when open floor plans with foosball tables and Keurigs were the key to unleashing workplace productivity restrained too long by dehumanizing cubicles? After enough writeups in even mainstream outlets like CBS News, everyone was doing it.

But we ended up just using headphones to fashion our own mental cubicles so we could work in some kind of awkward peace—particularly awkward
for women, people of color, introverts, and many others.

A similar thing happened with tech interviews. After all, boring accounting firms ask about your resume but not us! The media were awestruck at
profound interview questions like "Why are manhole covers round?" or "How would you calculate the number of cars passing through a busy bridge?"
or the famous Elon Musk riddle "You’re standing on the surface of the Earth. You walk one mile south, one mile west, and one mile north. You end up exactly where you started. Where are you?"

This is all about getting deeper than any traditional interview could, you see, to reveal how candidates think!

Yeah, OK.

Coding interviews are the worst. How do we know?

While the more absurd questions have lost some luster, one relic of our unconventional interview style that endures is the coding interview. You know it. A panel of dudes all vaguely reminiscent of Howard from The Big Bang Theory asks candidates to go to the whiteboard or a laptop broadcasting to a screen and write code for something they've probably never seen (by design). It could be implementing a doubly-linked list or merge sort or a function that determines if a string is a palindrome. It could be performing a breadth-first search or converting a loop into a recursive function. And on and on.

This is a problem—not only for job candidates but also for the companies considering them. It's an especially acute problem now with the shortage of software engineers and the popularity of remote work particularly in the aftermath of a global pandemic. Then there is all the uncertainty. Uncertainty tech companies feel in an era of inflation and evolving economic fundamentals and uncertainty candidates feel in an era of layoffs and evolving personal responsibilities and professional goals.

Coding interviews solve for the Computer Science Department valedictorian or the most short term knowledge. Not the best long term culture fit

The biggest technical problem with coding interviews is the disconnect between what they measure and what actually adds value to the business. For 99% of you, hiring someone who understands Kruskal’s algorithm for finding a minimum spanning tree isn't relevant to making your UI accessible, scaling in the cloud, improving your engineering practices, or driving revenue.

Even if you do better than silly college computer science questions and focus on the tech you use today, you focus on the wrong thing. It isn't what candidates know; it's how much they can learn. If you're a React shop, it's impressive if a candidate understands how useEffect is about syncing UI with state rather than a new spin on componentDidMount, but what if you decide to move to Solid? Or what if an important new initiative demands a different set of skills entirely and you don't have the resources to hire a subject-matter expert right away?

Instead, you want to hire engineers who fit with your culture and make their teams better with hard and soft skills. They do this is many ways. They learn quickly because think in abstractions so they can translate prior knowledge into new skills that serve their professional growth and your bottom line. They aren't afraid to experiment and fail in order to learn new things. They listen. They are eager to help. They are patient and kind. They manage their time well. They can communicate with customers. They care about good code and write good documentation. Their teammates trust them to support them and to assume leadership when the moments calls for it.

Your business needs Vision. Coding interviews at best get you Ultron.

Coding interviews further exclude systematically excluded communities

I am a passionate advocate for diversity in tech. Prejudice against anyone on the basis of race, gender, ethnicity, nationality, religion, orientation, ability, or even academic background not only limits our individual growth but also
limits the creative energy necessary to develop the best software. Sadly, there are countless institutional barriers to diversity in tech, and coding interviews are one of them.

Don't believe me? Check out this study by North Carolina State University and Microsoft:

But the format may also serve as a barrier to entire classes of candidates. For example, in our study, all of the women who took the public interview failed, while all of the women who took the private interview passed. Our study was limited, and a larger sample size would be needed to draw firm conclusions, but the idea that the very design of the interview process may effectively exclude an entire class of job candidates is troubling.

Much as society writ large is convinced DNA evidence and AI are "objective," tech is at best convinced coding interviews are also free from bias or at worst happy to capitalize on their bias. Bias in coding interviews could be favoritism, intentional or otherwise, reflected in choice of coding problem or leniency in evaluation. They are also biased against realistic forms of working. In real life, we don't have people staring at us as we type to scrutinize our work and mannerisms. We have access to Google, Stack Overflow, and other resources. The discomfort produced by contrived awkwardness in coding interviews hurts otherwise talented candidates—particularly those in communities already systematically excluded in tech like the women in the study.

But that's one study you say. OK, but consider anecdotal evidence from elite engineers. Jenn Creighton, senior engineer at Netflix on the NodeJS team and host of the single-threaded podcast exposed the problem on
her premier episode with her guest Erin Fox. Senior engineer, Apple whistleblower, and activist Cher Scarlett described her own experience a few years ago:

not work, and wasn't what they were looking for. These same two pieces of code are now written about by two other individuals as best practice solutions to these problems. Do you think this coding interview was about code? No, it was an ego battle I'd never be allowed to win.
— Cher Scarlett (@cherthedev) May 22, 2018

Of course if things can get so bad for cis white women in tech that coding interviews become a vanity exercise at their expense, just imagine how much worse it must be for women of color, trans people, anuerotypical people, even people who learned tech at boot camps instead of MIT or have degrees in psychology. And so many others.

Let's use the interview process to welcome as many people as possible into tech because the challenges that lie ahead are too great to leave anyone behind.

Coding interviews are the basis for an entire cottage industry dedicated to help candidates "ace" them

Precisely because coding interviews are so divorced from the reality of our lives as software engineers, job candidates need time to prepare for the experience. It takes time to build composure as people stare at you at the front of the room or judge every keystroke. It takes time to remember, or learn for the first time, fundamental concepts of computer science or low-level implementation details typically abstracted from you by frameworks and open-source libraries. It takes time to memorize it all because you may not be allowed access to the resources you normally have at your disposal.

Everyone knows it, so a cottage industry has emerged to get candidates through it. There are literally hundreds of books and online courses promising to help them "ace" the tech interview and get that prestigious, high-paying job so you can get that Cybertruck because you find its ugliness endearing.

This is costly. At best, it takes time away from personal life. At worst, it costs serious dollars. Either way, this is another way coding interviews serve as a gatekeeper excluding communities from tech—software engineers who can't afford the costs of interview prep.

What's the alternative?

Coding interviews are bad for the business and bad for the candidate. Instead, you should conduct interviews that are inclusive, find the best long term fit for your business, and convey the joy of solving problems there.

Make the interview a conversation. Share your business goals and details about your mission, the role you're hiring for, your expectations technically
and culturally, and your values as an organization. Then consider questions like these.

"What are some examples of problems you solved at work? Take me through how you solved them."

This question helps you understand the experiences candidates have had on the job. It helps you evaluate how these experiences apply to your business and, more importantly, how your candidates approach challenges. Feel free to explore the topic in depth and follow threads of conversation as candidates raise points you find particularly interesting. You don't have to be Oprah, but there is an art to navigating a conversation to elicit more information. You will get a good sense of candidates' technical knowledge, their creativity,
amd their ability to gain new knowledge.

After all, game recognizes game.

"Can you describe a situation at work where you were particularly proud of what you accomplished?"

This question helps lighten the mood and bring positivity into a tense process. It gives candidates a chance to brag a little but in a way that lends value to you. You can glean if your own business offers the kinds of opportunities for joy the candidates want to experience, and you can again get some detail into their experience, expertise, and potential for growth.

"It's common to have differences of opinion on a team on how to solve problems. How have you worked through those?"

This question addresses the reality of working on a team and the likelihood of a culture fit. Anyone who spends about three minutes on Tech Twitter knows we like to argue—often about the most trivial, self-serving things. Candidates have certainly had their share of disagreements at work, and it's important to understand how constructively they handle them and if their solutions fit with your company culture. You will also get a sense of what candidates think is important—tech or otherwise. The question is if your value system is compatible with theirs.

"If you had full control of the architecture and engineering of your current code base, what would you change about the code or the process?"

This question again helps you learn about a candidate's expertise and room for growth but from a different angle. We all have regrets about technical debt, yearnings for different approaches, and other aspects of the product and process that stick in our craw. Give candidates a chance to vent. In the process, you will learn about what candidates value, their knowledge of the engineering and product delivery landscapes, the direction they want to go, and their tact with constructive criticism. Does their vision comport with yours for your business?

"What kinds of support do you need to do your best work?"

This question addresses your responsibility to your people to achieve the goals of the business and the support candidates can expect if they join. You might have an issue with this question because it implicitly undermines conventional wisdom that the best engineers do their best work of their own accord if they have "passion" for tech implied by a robust GitHub presence, lots of side projects, eagerness to work weekends, etc.

Nope.

For reasons why the responsibility is yours to provide "flow state" rather than candidates' responsibility to have passion, readthis excellent post by Sarah Drasner, Director of Engineering, Core Developer Web at Google.

The best engineering managers understand how to create flow state. Even if candidates are unfamiliar with management theory, this question informs them that you have their best interests at heart and recognize your own responsibility in making them successful. You can also evaluate whether their answers are aligned with your business values.

"Do you have any questions for us?"

Hopefully you are asking this question anyway regardless of whether coding interviews are part of your hiring process. Give candidates a chance to ask you about the things that matter to them as they consider making the considerable, potentially life-changing commitment to your business. They should direct the conversation here. Expect questions about titles, commitment to and evidence of diversity, opportunities for advancement, remote work and tooling, and other aspects of business process and company culture.

Be careful though not to interpret the absence of questions as apathy or really anything negative. This is their time, and if they don't need it, you shouldn't let that influence your decision one way or another.

As an aside, prefer diverse panels of interviewers. Also pay attention to the demeanor of candidates. Any toxic behavior like arrogance, abruptness, or something particularly awful like refusing to acknowledge female interviewers should make your decision easy.

But OK, if you must do coding interviews...

Let's face it. Many of you will never accept that coding interviews are the worst. The idea is simply too entrenched in tech. You're convinced instant recall of computer science fundamentals is critical to bring value to your business. Besides, tech is full of imposters! You must expose them with cleverly chosen coding exercises they have never seen lest they compromise your business with their duplicity.

There is no evidence any of that is true, but it sure feeeeeeels true! The funny thing is there are actually a lot of job candidates who also think it's true. They see coding interviews as a perfectly reasonable expectation and happily, if sometimes nervously, prepare accordingly.

Fine. If you must, at least adapt your coding interviews like this:

Offer candidates the option either to live code or to do a short, paid(!) project to do at home.
Have them solve a real but non-critical and non-sensitive problem you have at work.
If it's a live coding exercise, collaborate with candidates rather than sit back and watch and give them access to resources.

We have a lot of important work ahead of us as an industry. I hope you consider the lessons I've learned to help your business grow to meet the challenges ahead.

Which Programming Languages Should I Learn First?

Neil Chaudhuri — Tue, 01 Feb 2022 00:12:09 +0000

People who want to get into programming often ask me which languages to learn first, but as Dee tells Naomi on the series premier of her eponymous TV show, "I can’t give you answers when you’re not asking the right questions."

I think it's more helpful to identify which problems you want to solve first. Do you want to build a website to sell your mom's amazing ghee? Look at Shopify, Squarespace, or Wix for a low- or no-code approach or Next.js or Remix Run for a bespoke site. Do you want to build a data model that can help American football coaches decide whether they should go for it on 4th down? Look at Anaconda or Tensor Flow Lite.

Don't let me stop you if you're jonesing to learn a particular programming language. But if you take my advice and come at the learning process from the point of view of providing a real business need (Your own business need counts!), figure out which tools, libraries, and frameworks make the most sense as you try to deliver for yourself and learn something along the way.

Secure Online Voting is Possible. This is How

Neil Chaudhuri — Tue, 25 Jan 2022 01:36:18 +0000

The right to vote is sacred. It's essential to live in a free society and to pick a winner on [The Voice (https://www.nbc.com/the-voice). Unfortunately, the right to vote is under attack worldwide, and bad actors have used tech to do it. Russia interfered with the Brexit referendum in the UK and with elections there, Ukraine, France, and famously here in the United States, even still as I write this, by hacking voting machines and voter registration databases and manipulating social media. Other foreign state actors are interfering in elections in similar if not quite as aggressive ways.

Meanwhile, within our own borders here at home, the state legislature in North Carolina used data science with "surgical precision" to rig state elections at the expense of voters' constitutional rights. North Carolina is hardly alone. In 2018 in Georgia, the Secretary of State, who was running for governor, had the authority to oversee his own election, and he orchestrated a complex web of corruption that actually encouraged security vulnerabilities and culminated with his own breathtaking wipe of election servers and backups.

If accountability bothers these politicians so much, they should work in meteorology or sports talk radio.

Even worse, the courts, including the Fifth Circuit Court of Appeals in Texas and the Supreme Court, have refused to remain above politics as the judiciary should and have chosen instead to legitimize these bad actors.

We can't forget criminal mercenaries either as we also saw when Microsoft took down Trickbot, in their attempt to infect voting infrastructure with ransomware.

The goals vary. To sow chaos. To sow mistrust of the electoral process. To override the will of voters. To preserve power structures. To change American policy objectives. To make money.

Regardless of the purpose of these attacks on the right to vote, they're shameful, and systemic flaws make it all possible:

Lack of transparency as manufacturers with troubling conflicts of interest produce closed, proprietary voting machines
Lack of knowledge or interest in modern software architecture patterns, engineering best practices, UX strategies, modern technologies, accessibility, and good security hygiene
Lack of training for staff in using and maintaining the available voting technology
Lack of funding for officials acting in good faith to build robust voting infrastructure
Lack of access controls to prevent officials acting in bad faith from exercising undue influence over the infrastructure for nefarious purposes
Lack of standards from NIST, DHS, and other government institutions for voting infrastructure

And many others. You can read about the threats to and core values of a secure online voting platform defined by a panel of experts.

Because of these systemic problems, the high stakes involved in electing officials who will be making life and death decisions in the era of COVID, the breathtaking incompetence and embarrassing failure of "modern" voting apps like the IowaReporterApp, and the fear of relentless attacks from around the nation and the world, we are in a place now where the consensus is that the only secure way to vote is with paper ballots. To say nothing of the vulnerabilities associated with paper ballots. Or the matter of voter access
to those paper ballots, which itself is fraught with bad faith arguments and racism.

It's easy, if profoundly disappointing, to see how we got here, but it really doesn't have to be this way.

Features of Modern Voting

There is an old joke:

I told the doctor it hurts when I do that. He said "Then don't do that!"

In order to build a modern voting solution, we need to look at the problems with existing voting solutions and "not do them."
I believe secure online voting must have several critical features.

Open Source (as much as possible)

In a world of absurd conspiracy theories and legitimate conflicts of interest for voting machine vendors, the core of the solution must be open-source. Every line of code needs to be available for all stakeholders—media, elected officials, election attorneys, security analysts, and most importantly voters—to have confidence in the security and integrity of the software and therefore in the outcomes of elections.

We will see shortly that it may be necessary to incorporate some commercial solutions that are closed-source, but those should be on the margins. There should be no core functionality that isn't available for all to see.

Another reason to value open source is that a mission as important as this demands diverse experiences and perspectives—on voting experiences, past software failures, biases, architectural patterns, tech stacks, potential vulnerabilities, and whatever else we need to understand to build the best online voting platform possible.

Zero Trust

Every state election system in America is perimeter-based. It's all about firewalls. Perimeter-based security is flawed because once intruders get in, which as we've seen time and time again isn't exactly impossible, there is no stopping them. Instead, we need a Zero Trust approach to security. I will let Chris Gerritz explain what this means:

Rather than defending only a single, enterprise-wide perimeter, the Zero Trust approach moves this perimeter to every network, system, user, and devices within and outside the organization. This movement is enabled by strong identities, multi-factor authentication, trusted endpoints, network segmentation, access controls, and user attribution to compartmentalize and regulate access to sensitive data and systems.

Rather than try in vain to erect barriers to compromise, Zero Trust assumes you have been compromised. This posture demands authentication and authorization at every point of interaction. It's more work, but it's necessary to minimize vulnerability to modern, sophisticated attacks.

Think of Zero Trust like a secure hotel. You need your key to get in the building, to get through front desk security, to take the elevator to your floor and only your floor, and finally to enter your room or the gym or the pool. Even inside your room, you need to be authenticated to access WiFi or use the safe.

Advancements in technology make Zero Trust possible, and a modern voting solution will enforce Zero Trust to ensure that every interaction with every component of the architecture demands authentication and a thorough vetting of privileges and integrity.

Transparent and Auditable

While the source code will be fully transparent to give everyone confidence in the integrity and fairness of election outcomes, we also need that transparency to extend to the runtime operation of the software. We need to know the health of the system and to know every single thing that happens throughout the architecture—and who did it. This means continuous monitoring throughout the stack, elegant visualizations of the telemetry, and if we can manage it, anomaly detection through analytics. This level of auditability is necessary for Zero Trust, and it is also particularly valuable for risk-limiting audits conducted after elections to assess their integrity.

Immutable and Append-Only

I have written and spoken a great deal about the value of immutable data, and I think it is essential for secure online voting. The software should not permit updates or deletes. Rather, any change to the data—a newly registered voter, a new address for an existing voter, and certainly every vote—should be represented in immutable, append-only fashion. As part of the auditability of the software, we should be able to replay every event to recreate state at any point in the process.

Client Device and Application Deployment Agnostic

A modern voting solution needs to give voters the freedom to access their ballots on a wide array of devices and to give state government officials the freedom to deploy on premise or in the cloud. The software needs to be agnostic to these possibilities, and this will force compromises in the implementation. It may not always be possible to apply the "coolest" solution if it couples the software too tightly to a particular vendor or feature.

Usable

This might seem obvious, but secure online voting demands all stakeholders consider it intuitive. This manifests in several ways. User interfaces need to reflect modern UX principles so that voters, poll workers, and state officials across age, education level, ability, and other factors find the software intuitive.

I also believe giving voters the opportunity to cast their ballots from wherever they wish is a form of usability. It's not just about comfort either. Because of the relentless media obsession with conflict, everyone tends to focus on candidates, but what about referendums, state constitutional amendments, and bond issues? These can be complex. I know from experience that it's helpful to research these comfortably from your home rather than under pressure within the space and time constraints of a traditional voting booth.

Part and parcel with usability is performance. Monitoring will help uncover issues with performance, but a modern online voting system needs to be architected for performance. Performance issues will not only be annoying, but they could also undermine confidence in the integrity of the vote.

Simple

To achieve all of this, there will be a lot going on in a secure online voting platform—user interfaces, APIs, encryption, databases, multifactor authentication, monitoring. It will be tempting to add complexity to integrate it all, and we need to resist the temptation to overengineer. Otherwise, the application will become unsustainable for maintainers and, much worse, unusable for voters and others, which will bring us right back to the status quo of a voting platform that diminishes the confidence we have in the integrity of our elections.

None of these are sufficient on their own. For example, merely being open source isn't enough to make this platform secure.

This is a lot, and in order to achieve it, a secure voting platform needs to be engineered with a continuous deployment
model that automates testing (for functionality, security, performance, accessibility), static analysis, and deployment. We need a process that solves for the key metrics for software delivery performance that Google describes in their State of DevOps report: deployment frequency, lead time for changes, time to restore service, and change failure rate.

The good news is that a secure online voting solution doesn't have steep demands for scalability or performance. It's not like there will be tens of thousands of votes per second. This isn't The Masked Singer.

Even if there were one instance of the platform for the entire United States, that's about 150 million voters. That's not a lot. And because in America every state runs its own platform, for better or worse, you'd have at most one instance of the platform for each state and inhabited territory. California is the largest state by population, and a statewide election there will have around 30 million voters. As software scale goes, 30 million over the course of several weeks (as the concept of "Election Day" thankfully grows more and more quaint) really isn't that much. Now maybe, hopefully even, great voting software will raise those numbers, but as it stands now, this makes things a lot easier. We can focus on user experience, data integrity, and platform security and worry a bit less about performance at scale.

Another bit of good news? The UI is simple. It's just boring forms!

What Might the Tech Stack Look Like?

I'm not exactly sure, and there are a lot of great options. But I do have some ideas I would like to run by you.

The user interface: Remix Run PWA

Remix Run is a new open source web framework from the creators of React Router that provides abstractions over core web fundamentals to build a resilient experience. In fact, it's that resiliency that makes Remix a compelling choice for a voting application. It's lightweight because it relies on core browser APIs and HTTP, and forms still work without any JavaScript because HTTP supports form submission on its own.

Certainly a front end in Rails or another robust monolithic framework would be effective as would alternatives like Next.js or SvelteKit, but I find it hard to resist Remix's lean philosophy. As part of the trend towards more SSR at the edge, Remix Run even offers templates for a variety of deployment platforms like Cloudflare, Fly.io, Netlify, and Vercel, and that list will only grow. Although edge functionality isn't essential for voting software that applies across such little territory (a single state), the deployment flexibility is essential, and every millisecond helps.

In addition, I think it is important that the front end is deployed as a Progressive Web App. This offers a lot of benefits, but primarily for this purpose, it is critical that the front end is always available and as functional as possible regardless of connectivity, which absurdly remains a problem in the richest country in the world.

This is orthogonal to the choice for UI or PWAs, but the voting application needs to have Cross-Site Request Forgery protections and a strong
Content Security Policy as well.

By the way, it might be interesting down the road to think of voice interfaces allowing people to vote with Google Assistant, Alexa, or Siri if privacy concerns can be addressed. One challenge at a time though.

Database: PostgreSQL. With a twist

The backbone of this architecture is the event sourcing architecture based on an immutable, append-only data store representing every single mutation to the data on the platform in order to ensure full replayability and traceability for risk-limiting audits. How can we do this with PostgreSQL?

Easy. Revoke UPDATE and DELETE privileges!

Anything more than PostgreSQL, which is straightforward to deploy and agnostic of environment, would be overkill given the small scale—particularly if someone deploys the online voting platform for a small election below the state level with just a few thousand or even a few hundred voters.

We could store votes in a single table where a simple GROUP BY will aggregate election results. That's easy. We can also store temporal and location data so we can run some basic secondary queries like measuring voter activity by precinct or time of day or day of week or whatever else you want to know. Changes to voter information, a popular action for bad actors, would also be tracked as immutable events, and voters would be notified as they happen.

So immutable PostgreSQL it is. It's easy to connect to PostgreSQL from the UI with Prisma.

By the way, what about blockchain? No. Just. No.

Deployment: Somewhere easy

I have no particular preference for where we deploy this online voting platform, but it needs to be somewhere offering good DX and resiliency. To me, this implies any of the out-of-the-box deployment targets for Remix Run, but it could be AWS, Heroku, or many other cloud providers. There could also be a combination like Remix Run on Cloudflare and PostgreSQL, SMTP, and other infrastructure on Supabase. It all depends on what's simple, cost effective, and meets the needs of voters and staff.

Monitoring, Auditing, Disaster Recovery, and High Availability: It depends

An online voting platform will come under attack from the most sophisticated hackers in the world, and its single most important requirement is that it always maintains the confidence of voters. In order to face these challenges, we need monitoring, auditing, disaster recovery, and high availability.

These are cross-cutting concerns that apply to any non-trivial deployment, but they are undeniably essential here. It's hard to identify particular solutions because they are a function of the deployment platform, but suffice to say that any platform that cannot meet the needs of a mission-critical system like anomaly detection, alerting, and consistent backups is a nonstarter.

In the end, we need to be able to understand who, what, where, when, and how for every single event on the platform.

Authentication and Authorization: A blend of open-source and proprietary solutions

It goes without saying that the most important piece of online voting is security. The challenge isn't just technical though:

It needs to be simple to implement and maintain yet all but impregnable
It needs to be simple to use by all voters regardless their age, ability, tech savvy, and other factors
It must withstand independent audit by a trusted partner

This is where the top minds in infosec will be invaluable. I am nowhere near that class, but let me throw out some ideas.

In the interest of Zero Trust, the connections to the server, the database, and any other infrastructure like SMTP servers and caches will be authenticated over TLS, and all data at rest will be encrypted. This of course implies encryption key storage like that provided by HashiCorp Vault and similar products.

Voters would have a choice of authentication methods:

Username/password (12-64 characters with mixed case, numbers, and special characters required) and their choice of MFA methods (e.g. authenticator app, physical key) along with the usual Forgot Password, Change Password flows
OpenID integration with Google

There will always be voters who feel comfortable voting in the traditional way—showing up to their local polling places on Election Day and casting ballots. Polling places would simply be equipped with computers from which voters cast their ballots by setting up accounts and logging in through the browser to our secure online voting platform, and staff would assist in the process. If it were up to me, the government would provide funding for physical keys, and every voter who shows up in person on Election Day who wants one would get one and would be shown how to use it.

The database could associate random tokens to users, analogous to an access key for a cloud API, which for voters would have a quota limit of one for the duration of eligible voting. These tokens would be encrypted and rotated on a periodic basis, and they would represent voters in their interactions with the APIs to cast votes and staff in all other API calls.

Finally, securing the entire DevSecOps pipeline means implementing a host of measures like keeping secrets out of code and configuration, managing access control and limiting permissions throughout the pipeline, signing changes to version control with PGP, using key management mechanisms appropriate for the deployment platform, and much more.

And of course there are automated security tests in CI and full audits by security professionals to vet the entire
architecture.

This entire stack, and really the whole architecture, is just an idea. It is all subject to change.

Outstanding Questions

Even if the architecture and technology stack are perfect, there are difficult questions that remain across not only technology but also law, finance, politics, and even philosophy. Here are some of them:

Every state has its own election laws, technology infrastructure, and budget. What kinds of legal, privacy, and technical challenges are there to migrating voter registration data to a new system? Is there even a need if the application can represent registered voters some other way?
Corrupt officials do not want anything that will make voting easier, but would even honest officials consider it?
What's to stop government officials who are authorized users acting in bad faith from compromising the platform in some way?
While the platform would be built for resilience, what kinds of contingency plans would be in place just in case the platform went down for an extended period?
If we use PostgreSQL as an immutable, append-only store to provide a replayable log of all data mutations, we will eventually hit its limits. What's the retention period for the data? If it is even necessary to retire the data to some kind of data lake after the retention period, where would that be? How would that work?
To what extent can we preserve the notion of a "secret ballot" where only voters themselves know their selections? Or should a modern voting platform recognize the very concept of a secret ballot as an anachronism that is pointless at best and harmful at worst and function accordingly?
Would machine learning serve a purpose here? If so, what's the simplest and most effective way to implement it?
Would there be an audience for making non-PII data available via APIs for data analytics by independent organizations? If so, how would we do that?

The beauty of open source is the diversity of thought and creative energy that converges to solve interesting, hard problems like these.

It will take a historical effort to build a secure online voting platform that allows all registered voters to make their voices heard and gives them the confidence that their votes count and that winners are legitimate. If you find improving access to voting, guaranteeing the integrity of elections, promoting social justice, and solving interesting problems as important and as compelling as I do, please get in touch so we can collaborate on something that could transform society for the better.

What If...Marvel Built a Minimum Viable Product?

Neil Chaudhuri — Thu, 11 Nov 2021 01:50:46 +0000

In 2011, Eric Ries published The Lean Startup, and it revolutionized software development. Lean principles had already made it into agile software development, but The Lean Startup fills in a lot of gaps to turn the abstractprinciples behind the Agile Manifesto into something real.

For example, even if you nailed agile and built the thing right, how do you know you built the right thing? In other words, it's great to execute on your project and build exactly what you intend on time and on budget, but when you're done, how do you know people will actually buy it? Although the Agile Manifesto never answers this question because it assumes you are on the right track all along, Ries gives us an answer.

The Minimum Viable Product

Probably the most profound innovation in The Lean Startup is the Minimum Viable Product (MVP).

(You can watch Ries himself talk about it, but I have to warn you that it looks like it was filmed by the director of The Blair Witch Project.)

The idea is really quite simple. Whether you are conscious of it or not, the product you have in mind is based on your assumptions about what your customers want and why, so the purpose of the MVP is to validate those assumptions with hard data around a lightweight representation of your product before you commit your full resources to building it out in full. You give your early adopters access to the MVP and consider their feedback, which Ries calls "validated learning." Validated learning is crucial. For Ries, it's quantifiable feedback that is "the unit of progress for lean startups."

Based on what you learned, you either "pivot" to something else more aligned to what your customers want or "persevere" because you had it right all along. From this point on, you have quantifiable, proven demand for your product, and you can be confident that you will be investing in building the right thing.

Like a lot of terms in tech that blow up, consultants often use MVP to mean something different from what Ries intended--most commonly to mean the first version of your product and/or a really scaled down version of it. In either case you've implicitly decided without any validated learning what you want to build. The fact it's in its earliest stages is irrelevant.

So what does all of this have to do with Marvel?

The MCU Juggernaut

Unless you have been living in the Quantum Realm for the last decade, you know the Marvel Cinematic Universe (MCU). It's a monumental achievement in cinema that has taken decades of Marvel comics and reformulated them into a multibillion dollar juggernaut movie franchise that has raised once second-tier heroes like Iron-Man, Thor, and Black Widow to the stature of eternal favorites like Spider-Man, Captain America, and Hulk and put Mjolnirs and Infinity Gauntlets into the homes of ardent fans worldwide.

It may be hard to imagine now, but there was no guarantee of success at the beginning as Marvel was struggling as a business having sold off their most bankable assets. This forced Kevin Feige, the architect of the MCU, to get creative to and derive new ways to bring Marvel heroes to life.

They succeeded spectacularly, and now they have to do it all over again.

Firmly entrenched in global pop culture, the MCU now faces pressure to build on its foundation with new heroes and villains, many of whom like Moon Knight and Titania lack the star power of Captain America and Spider-Man and are all but unknown to mainstream audiences. And considering that the MCU has decided to go all in on the Multiverse, there are literally infinite possibilities for stories.

How can Marvel decide what direction to take with the MCU?

What If? is Marvel's MVP

There is no question Kevin Feige has sketched out the broad contours of the next iteration of the MCU. Still, as sterling as his record is, he's working from assumptions about what will make for great stories and what will thrill audiences going forward. If he followed The Lean Startup model, he would build a lightweight MVP to test his hypotheses with passionate fans and measure their response for some validated learning.

It turns out that is exactly what Marvel is doing with What If?...

What If?... is an animated show on Disney+ that imagines the MCU heroes we know experiencing very different lives throughout the multiverse. For example, we see Peggy Carter receive the super soldier serum rather than Steve Rogers to become Captain Carter, and we watch somberly as T'Challa, voiced just as he is portrayed on screen by the legendary Chadwick Boseman in his final performance, becomes Star-Lord rather than Peter Quill. It's heroes and villains we know so well from the MCU facing new challenges alongside other heroes and villains we've never seen them interact with before.

The novel combinations are fun and exciting, but they're also an experiment. How will fans, primarily hardcore MCU fans (early adopters if you will) react? You can easily track this with viewership metrics on Disney+, mentions on Twitter, and the impressions of media influencers for validated learning. When a story doesn't click, Marvel can dismiss it as a fun idea for the fans and pivot to something else. On the other hand, when a story does click, Marvel can persevere and explore it further on a live-action series on Disney+ or even on the big screen within established MCU canon.

Best of all, Marvel can test their hypotheses, engage in validated learning, and guide the future of the MCU accordingly at a low cost. What If?... is high-quality animation, and most of the elite talent that portrays the characters on the big screen voices them on the show. Still, the cost of running these experiments in animation without any real commitment since the stories all happen outside the "prime" MCU universe is orders of magnitude less than it would be going all in with any of the What If?... plots in live action.

The Key Lesson

One way or another, we are all in the business of product development. Too often we assume we have accurately gauged customer sentiment, and we exude hubris that success is inevitable only to fail miserably.

Don't be afraid to test your assumptions before you go all in. Build a Minimum Viable Product--something more lightweight and less expensive that will nonetheless provide your early adopters something meaningful to evaluate. The validated learning you get from their feedback will either help you pivot to something more aligned with what people want or empower you to persevere with your original idea with legitimate confidence it will sell. Once you start executing, then you can argue on Twitter about whether you need daily meetings and whether people should be allowed to sit down in them.

Using an MVP to build confidence in your product direction is exactly what the Marvel Cinematic Universe has done with What If?..., but you don't have to be Marvel to fly higher, further, faster, baby.

Lessons Learned from Building a React Component Library with TypeScript

Neil Chaudhuri — Tue, 19 Oct 2021 20:47:56 +0000

Component libraries are all the rage. Shopify, Salesforce, IBM, and even the United States government have joined countless other organizations and businesses in building component libraries. They're the subject of blog posts, podcasts, and YouTube tutorials. All that's left is a Ken Burns documentary on the subject.

In fact, I am a software architect and senior engineer, and I currently lead the development of a React component library that will be the basis for the UIs for a prominent US government agency. I want to share with you my lessons learned in project management, communications, accessibility, engineering, and testing to build something that will impact the lives of millions. And the ups and downs of it all.

So what's the big deal with component libraries?

The Design System

It doesn't start with a component library; it starts with a design system. The Nielsen Norman Group defines design systems this way:

A design system is a complete set of standards intended to manage design at scale using reusable components and patterns.

A design system enumerates the standards and practices that comprise the premier UX for consumers of your brand. It expresses the nomenclature every team should use in communications to break down silos and avoid the impulse from Conway's Law. There are basic rules about colors, typography, spacing, and so on. All of these core principles become the basis for larger components--explicit ones like buttons and date pickers and subtler ones like grid systems.

Our UX team develops and maintains our design system. Like software, it evolves; it's versioned; and it's collaborative. There are conversations among the UX designers and with me and other architects and engineers on the program about what makes sense and what is feasible. Are nested dropdowns necessary? Do we have time to create our own perfect Datepicker? Or do we try to customize something open source? How do we feel about disabled buttons, and if we think they make sense, how can we overcome common pitfalls like poor contrast ratios?

Stuff like that. We use the language of Atomic Design, which deconstructs web interfaces into entities ranging from "atoms" to "pages," as a common nomenclature to describe the goals of the design system.

The challenge, and probably the hardest part of building a component library for us, is the tooling. Partly because of the preferences of the UX team and partly because of constraints on our development environment due to the sensitive nature of our work, we have not been able to streamline automation for versioning UX wireframes or translating them into artifacts engineers can use to build. As a result, we work with wireframes that are cumbersome to understand. In order to even view them, we either need to install the tool on our machines, which costs more licenses and imposes a burden on developer experience (DX), or we need to wade through literally hundreds of static asset files with a custom browser plugin. Neither is an optimal experience. Beyond that, it's a manual process to track consistency between the design system and the component library as both evolve.

I never said it was pretty, but it isn't all bad either.

The Value of a Component Library

The design system is a set of core principles independent of implementation details. You can choose to implement these principles and make them real for UI engineers with whatever technology you choose.

For us, that's React. Our React components generate a lot of value for the program.

Consistency

Our component library enforces our design system across our development teams. Using the components all but guarantees a UI will be consistent with our brand and provide our users the best, most intuitive experience. Developers can feel confident they are using components vetted with the UX team, which frees them up to work on the specific use cases of their services rather than cross-cutting concerns like consistency with the design system.

The library also maximizes the likelihood that our UIs pass visual testing by our UX team. This is important as violations slow down our delivery cadence and ability to get feedback.

Accessibility

Related to consistency is accessibility, which is a first-class priority for our component library. Accessibility, commonly known as #a11y, is more than just empowering the visually impaired. It also means empowering people who experience difficulty with hearing, motion, dexterity, or anything else. It means empowering everyone.

The program is required by contract and by law to produce UIs that
are accessible--specifically 508 compliance. That said, accessibility is far more than a professional obligation; it is my personal priority. It is very important to me that everything I build is intuitive for every user.

I will elaborate on this shortly, but our component library is built for accessibility. Development teams can trust the accessibility of the individual components, and as I said before, focus on their own use cases. Of course you are probably thinking in terms of accessible dropdowns and autocompletes and datepickers, which we have, but we also provide helper Semantic HTML components. For example, the library features Section, which represents the section HTML element as you would imagine, and SectionGrid, which is a section element endowed with our design system grid.

Of course, the component library can only take developers part of the way to full accessibility, but it's nice not to have to start from 0.

Reusability

We have worked very hard to provide intuitive APIs for our components, but the task is trickier than you might think. The APIs need to impose enough opinion so that consumers don't violate the design system but allow enough freedom for the components to support a wide range of use cases. For our Button component, that is easy enough. For layout components like Card and Page, it's tougher. The reusability that results has made individual teams and the entire program so much more productive.

We also go out of our way to endow our components with as little functionality as possible. Component APIs offer props that enable library consumers on the development teams to supply behavior. For an obvious example, developers supply onClick behavior to the Button component. We have more complex components that need to maintain their own state,
but we try to minimize that where possible. This provides a clean separation of concerns, which makes testing our components much easier, and anyone who has been in the game long enough knows that strong testability makes for strong reusability.

Encapsulation

There will be more about this shortly, but we do not build our components from scratch. Rather, we customize existing open source components and map our APIs to theirs. This abstracts the implementation details of the component from our development teams. For example, we use react-datepicker as the basis for our own DatePicker, but if we decide to swap it out for a different one, our consumers will be none the wiser.

Component Stack

As I mentioned, we build our component library with React, which is what we recommended but is also, for our risk-averse government customer, the safe choice given its backing by Facebook, its market penetration, and its popularity.

But React is the easy part. Let's look at other parts of the component stack.

TypeScript

When we started building the component library, I considered TypeScript essential for two reasons. By enforcing type safety during development and at build time, we catch bugs much faster, which from a project management standpoint is much cheaper. More importantly, building our APIs in TypeScript is a huge help to library consumers on application development teams by facilitating code completion in their IDEs and type checking in their builds.

Let me also mention that some of our TypeScript APIs require ARIA values to promote accessibility if we can't derive them ourselves from other props.

Chakra UI

I mentioned earlier that our components are built on open source components, and most of them are built on Chakra UI. There are many other open source component libraries out there, but Chakra UI is my favorite by far. The primary reasons are its first-class commitment to accessibility and the intuitive APIs of its components built with TypeScript. As you can probably infer, Chakra UI is an inspiration to me when building our own
component library on top of it.

Chakra UI also offers a powerful theme customization API we leverage heavily to apply the principles of our design system to Chakra components via dedicated theme files that separate the styling from functionality. This separation of concerns makes it easier to reason about our code and makes the files themselves a lot lighter.

Chakra UI also features with some helpful hooks like useDisclosure that come in handy.

If you use Chakra UI for your own component library, you will probably need some alias imports to deal with name collisions. For example, we call our button components, to no one's surprise, Button, but so does Chakra UI. So we do this:

import { Button as ChakraButton } from "@chakra-ui/react"

Engineering

Of course the fun part is building a React component library. This post is long enough, so I can't get into every detail. But I do want to address some of the key aspects you might want to consider when you build your own.

Workflow

When we first began building the component library, we needed to move quickly because development teams were waiting on us
to start building their UIs. Our management tasked me and several developers to get something done in a few sprints at nearly a full time commitment.

We got the initial design system specification from the UX team and got to work. After those first few sprints, we had built enough components to allow teams to get going. The problem is that all of us resumed our normal duties with no time allocation for the library. This meant that whenever the UX team designed new components or developers found bugs in existing components, there was a bottleneck because no one was dedicated to upgrading the library. I and others got to it when we could, but the absence of a dedicated team was a problem.

Another problem is the initial lack of communication within the UX team itself and among the UX team, developers, and me. In their creative zeal, far too often they provided wireframes to some developers inconsistent with wireframes provided to others, or they provided wireframes featuring components that weren't in the library. Development teams assumed they were in the library and estimated accordingly. As you might expect, they were unhappy when they discovered the components didn't exist, which impacted their ability to deliver on schedule. They let me know it, and frankly they had every right to be unhappy. I knew we had to improve our process.

To that end, we made some changes. We established a Microsoft Teams channel to encourage communication by eliminating the ceremony of meetings and even E-mails. We also decided that development teams will build new components initially, and if other teams will benefit, the library will absorb them, with tweaks as needed to APIs or implementations, to support broader applicability across the program. Then the team that built the component first will replace their implementation with the library's when ready. While this means teams have to devote more time to developing components, it's transparent, and there is no bottleneck.

This is an evolving workflow. There is always room for improvement.

Component structure

Our components in TypeScript take three forms.

The simplest components look like this:

export const TimePicker = (p: TimePickerProps) => {
    ...
}

Our TimePicker component has no children, so it's as straightforward as it gets. It's just a function!

If the component has children, it still isn't too bad:

export const Card: React.FC<CardProps> = p => {
    ...
}

React's FC type (for FunctionComponent) includes a children prop implicitly. We could also declare it just as we do TimePicker but explicitly add a children prop of type ReactNode to CardProps. I prefer FC because it very clearly signifies the presence of children to library consumers and because the type parameter lets me enjoy some type inference. Notice how I don't have to specify the type of p because it's implicit from the type parameter CardProps.

Still, not too bad, right?

The last kind of component is a little complicated--form components. Our developers use React Hook Form, and like every other form library I've used, it uses refs to maintain form state. This means our components need to provide a way to accept a ref and delegate it to their children.

Most React engineers don't know this because they don't have to, but React provides a function for exactly this purpose called forwardRef, and we use it like this:

export const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(function Button(p, ref) {
    ...
}

Let me try to break this down.

A higher-order function is a function that takes functions as parameters or returns a function. Here forwardRef takes that Button function that renders the component as a parameter. Thanks to forwardRef, development teams can pass refs to the form components in our library, which we pass along though that function parameter to our rendered implementation. The type parameters to forwardRef provide type safety and inference. The type of p is ButtonProps, and the ref will be hooked onto a HTMLButtonElement.

In the end, it's a little complicated and a fair bit of ceremony, but the result is pretty simple--a form component that accepts a ref from the caller so form libraries can work with it as needed.

Directory Structure

When considering how to lay out your source code, it comes down to your team's preference, but as I tweeted recently:

There is a lot of commentary on how we should lay out source code in #React. If you take two "things" (functions, classes, #TypeScript interfaces, etc.), the higher the frequency that changing one changes the other, the closer they should be together
— Neil Chaudhuri (@realneilc) September 29, 2021

What does that really mean in practice?

Simple. When it comes to our component library, this means organizing code dedicated to a particular component in the same directory and even in some cases the same file. This is how we do it at a high level.

Our Button.tsx contains the ButtonProps interface, related types, and of course the component itself. Meanwhile, I love how Chakra UI allows us to separate theming from behavior, so the colors, spacing, font family, icon sizes, focus behavior, and other button details defined by our design system are in ButtonTheme.ts, a different file in the same directory.

Finally, although we could keep our tests and stories (more on these later) in the same directory, we prefer organizing them in their own subdirectories. I guess I've seen too much Marie Kondo.

TypeScript Config

I come from a background in statically and strongly typed programming languages like Java and Scala. While I understand longtime JavaScript engineers balk at types, I find types make me extremely productive. As a result, our TypeScript config is very strict. In particular from our tsconfig.json:

{
...
  "compilerOptions": {
    ...
    "noUnusedParameters": true,
    "noImplicitReturns": true,
    "noFallthroughCasesInSwitch": true,
    "noImplicitAny": true,
    ...
  },
...
}

As for building the library for application development teams, we scope our tsconfig.json this way:

{
...
  "include": [
    "src/**/*"
  ],
  "exclude": [
    "**/__stories__/*",
    "**/__test__/*"
  ],
...
}

All our components, stories, and tests are in the src directory, but we only want the components when we build the library. This is why we exclude the __stories__ and __test__ directories inside each component directory.

Static Analysis and Code Formatting

Like everyone else, we rely on eslint and Prettier, and we don't do anything particularly special. Still, I do want to mention a couple of things.

First is eslint-plugin-jsx-a11y. We use this eslint plugin to automate verification of the accessibility of our component library. It checks the JSX of our components for obvious violations. This is as far as we can go with automation, but we complement eslint-plugin-jsx-a11y with manual auditing in Storybook I will discuss shortly.

There might be something gnawing at the experienced engineers reading this. In the tsconfig.json above, we exclude our stories and tests because they don't belong in the build. Still, you know we should apply the same quality standards to story code and test code as we do to production code. Code is code.

To do this, we extend tsconfig.json in a file called tsconfig.eslint.json,
replacing the exclude field with an empty array, and configure eslint to use that. This tells eslint (and therefore Prettier) to include everything in the src folder in its analysis with identical TypeScript configuration. This means, for example, we can't cheat by using an implicit any in our stories or tests either.

Builds

We run our builds with Vite. That may seem counterintuitive since Vite is the build tool for Vue while our library is built with React, but Vite is actually agnostic. In fact, it amazed me how little configuration we needed. It basically just worked. Our Vite config is almost identical to the example in the documentation. Just like the example, our build produces two bundle formats--es and umd--and it works fast.

As you may know, TypeScript builds feature two phases, type checking and transpilation to JavaScript. Type checking by tsc, the TypeScript compiler, is very slow, so while it is very important, you should do it rarely. We only do it via the IDE in real time as we code or when we build the library for production--and break the build if type checking fails.

We have a dedicated typecheck script in our package.json that looks like this:

{
  "scripts": {
    ...
    "typecheck": "tsc --p tsconfig.eslint.json --skipLibCheck --sourceRoot src --noEmit",
    ...
  }
}

Note that we use tsconfig.eslint.json to typecheck everything.

Meanwhile, transpiling your TypeScript source code to JavaScript is faster than type checking, but so is reading Tolstoy. Transpiling with tsc or Babel is still not fast. However, the transpiler esbuild is written in Go, a language built for speed, and Vite uses it under the hood. Because we are transpiling constantly to see what's happening in Storybook, it's crucial that the process be fast. Thanks to esbuild, Vite does exactly what we need.

Our production build, versioned with Semantic Versioning, includes declaration files for each component and an index.d.ts file enumerating all components. These improve DX by enabling developers' IDEs to perform fast code completion. We also provide the theme file we use for our own components so that developers can apply the same theme to theirs. Our CI/CD pipeline publishes the library to a private NPM registry, which allows appropriately configured npm installations on developer machines to fetch the library with a conventional npm install. The package.json file accompanying the library contains all the peer dependencies they will need to use the library so npm can grab them, and for convenience it also contains the version of the design system it is built with for developers to track.

It also contains configurations to define which files to package in the library and how consumers can import modules:

{
...  
  "files": [
    "dist"
  ],
  "types": "./dist/index.d.ts",
  "main": "./dist/components.umd.js",
  "module": "./dist/components.es.js",
  "exports": {
    ".": {
      "import": "./dist/components.es.js",
      "require": "./dist/components.umd.js"
    }
  }
...
}

One last thing to note about the build. Although Vite of course provides minifying and other production readiness capabilities, we don't use them. We bundle the component library completely "raw." We find this helps developers debug their applications and report bugs (in those rare cases we make mistakes) with specificity. When they run their own builds, their tooling will apply minifying, tree shaking, and all other production processing to all their code and dependencies including the component library.

Testing

As I mentioned before, we limit the functionality of our components to the bare minimum necessary to add value. Still, components are code, and our consumers have expectations of our code. This means we need to test our components as much as we can and where it makes sense.

Testing is a controversial topic. On Tech Twitter, engineers are more than happy to let you know why you are wrong to test your code in a different way than they do. I can only describe what works for us and why we think so while also stipulating that our methods are subject to change as we get better at this.

Our approach is heavily inspired by this Storybook blog post. In it, Varun Cachar describes different types of testing, when each is appropriate, and which tools make sense for which types based on the experiences of several large-scale engineering teams.

Storybook

Storybook is crucial to the development and testing of the component library for us, and it's indispensable documentation for our users.

During development, we use it in a couple of ways. If the component is simple, then it's nice to have your code and Storybook side by side and watch your changes render as you make them with hot reload. On the other hand, when we aren't clear on what the API for a component should be, it's nice to write a few stories to work out the DX for it. Experienced engineers might recognize this approach as analogous to
Test-Driven Development (TDD).

We apply our design system custom theme in Chakra UI to every story in preview.jsx:

export const decorators = [Story => <ChakraProvider theme={theme}>{Story()}</ChakraProvider>]

During testing, we also use Storybook in multiple ways. For example, because we take a mobile first approach to our components, which matters for organisms in particular like modals, we configure custom breakpoints like this in preview.jsx:

export const parameters = {
    viewport: {
        viewports: {
            xs: {
                name: "XS",
                styles: {
                    height: "568px",
                    width: "320px",
                },
                type: "mobile",
            },
            sm: {
                name: "SM",
                styles: {
                    height: "896px",
                    width: "480px",
                },
                type: "mobile",
            },
            md: {...},
            lg: {...},
            xl: {...},
        defaultViewport: "xs",
    },
}

I mentioned a CI/CD pipeline that builds the library and publishes it to a private registry. It turns out the pipeline also publishes our component Storybook to an Nginx container so that the UX team can conduct visual testing on the components, and the ability to toggle among viewport sizes is extremely helpful.

It's also helpful for development teams who use our components to interact with them. Thanks to Storybook Controls, they can configure components themselves to see what happens. Thanks to Storybook Docs, they can see the code and API props that generate each story. So Storybook provides a profound documentation benefit throughout the program.

We also use Storybook for composition testing occasionally though not as often as the Storybook team may prefer. For example, we have stories that demonstrate how to integrate our form components with React Hook Form, and this exposed issues we had with our refs. Generally though, we don't do a lot of composition testing until we need to reproduce a scenario to fix a bug (and prove we've fixed it eventually).

We make heavy use of storybook-addon-a11y to test for accessibility. As you can see from another post by Varun Cachar, who is definitely earning his paycheck, Storybook offers a lot of features for accessibility testing. We make use of all of them. As I mentioned before, even though we do our best with jsx-a11y in the build and Storybook visually to test for accessibility, it is still incumbent upon teams to add @axe-core/react to their builds and perform their own visual tests in order to feel as confident as we can that we are providing the best possible experience to all our users.

Finally, while Storybook has been invaluable for us and I recommend it strongly, I would be remiss if I didn't mention some gotchas. Storybook uses a lot of the same libraries we all use for theming, Markdown, and other things. When there are library conflicts between your version and theirs, bad things happen. For example, we got hit with the same conflict on Emotion as this issue on GitHub. To its credit, the Storybook team releases frequently. If nothing else, make sure you use identical versions of Storybook and all its addons and that you upgrade as soon as possible when updates are available.

Storybook is also well aware of the "DivOps" revolution in JavaScript build tooling and is positioning itself accordingly. This is exciting since Webpack had a good run but feels more and more like the past, and we wanted to use Vite with Storybook. We installed storybook-builder-vite knowing it's experimental to see how it would work for us. Overall, it makes our Storybook builds fast just as we hoped. Still, when you consider storybook-builder-vite is raw, community-led by great engineers who have already given the community so much with their limited time and can't address every issue, and the general brittleness of Storybook I mentioned, your mileage may vary. Here is our Vite-related Storybook configuration in main.js:

module.exports = {
    ...
    core: {
        builder: "storybook-builder-vite"
    },
    viteFinal: async config => {
        return {
            ...config,
            plugins: ...,
            optimizeDeps: {
                ...config.optimizeDeps,
                entries: [`${path.relative(config.root, path.resolve(__dirname, "../src"))}/**/__stories__/*.stories.@(ts|tsx)`],
            },
        }
    },
}

React Testing Library

If you have read any of my posts on testing, you know that I think our industry writ large gets testing wrong. We test some things too much. We test other things too little. We don't always know the purpose of our tests. And worst of all, because of perverse incentives, we write tests to check a box.

I mentioned earlier that it has been a priority to endow our components with as little behavior as possible. Aside from the fact simpler code is easier to maintain and understand, this approach means fewer surprises for our consumers and less for us to test.

Or so I thought.

Our program has a mandatory minimum of 80% code coverage for our applications, and for reasons that don't make a lot of sense to me, that also applies to the component library. In my view, only components that maintain internal state offer the complexity that demands the ceremony of formal tests beyond Storybook, but alas, I don't make the rules.

React Testing Library has become the de facto standard for interaction testing in React, and of course we use it for our own tests. But how could we write tests as quickly as possible to limit the impact of the code coverage standard?

If you have written tests in any programming language, you understand the concept of "test fixtures," the setup for your tests. For us, that means test fixtures are simply components configured with different props.

But isn't that exactly what stories in Storybook are?

Storybook offers a feature I love--the ability to import stories into tests written with React Testing Library as fixtures using
@storybook/testing-react. Without it, we would need to duplicate
the same code as stories in Storybook and fixtures in tests. The autocompletion is great too thanks to the TypeScript support built into @storybook/testing-react.

One last thing I want to mention is, as you might guess given how much I have emphasized it in this post, accessibility. All of our tests in React Testing Library use getByRole and findByRole selectors. We do this because it is a way to build implicit accessibility testing into our interaction tests as the documentation describes. After all, if we are unable to locate the component we wish to test by its ARIA role, that all but guarantees it isn't accessible. And if it isn't accessible, I don't care if it "works" because it doesn't work for everyone.

Aside from all that, our tests work exactly as you would expect if you know React Testing Library. Here is an example of a simple test conveying everything I described:

...
import {
    DefaultMediumPrimaryButton,
    ...
} from "../__stories__/Button.stories"

test("Button primary display works", () => {
    const onClickMock = jest.fn()

    render(<DefaultMediumPrimaryButton onClick={onClickMock} />)

    const button = screen.getByRole("button", { name: "Primary" })

    userEvent.click(button)
    expect(onClickMock).toHaveBeenCalledTimes(1)
})

I know this is a lot, and it might have been slightly more entertaining as an audiobook. Still, I hope I conveyed the value in design systems and component libraries and the lessons we learned in project management, communications, accessibility, engineering, and testing to build something that will impact the lives of millions. I hope you can do the same...but better.

Now go take a nap. You earned it.

Scala or Go: Who Wore It Better?

Neil Chaudhuri — Tue, 07 Sep 2021 02:49:32 +0000

Scala and Go are two of the fastest growing leading-edge programming languages in the world. In the United States, they are also among the most lucrative. Scala and Go are among a slew of programming languages that innovate in numerous ways to produce faster, more resilient, more secure applications for a multicore, cloud native, mobile world.

The thing is Scala and Go have very different philosophies on what makes engineers most productive and what defines great applications. We are going to look at how Scala and Go solve five common programming tasks and how their contrasting approaches reflect their contrasting philosophies. Then you can decide for yourself which works best for your next application. This is a really long post, so here are the tasks we will consider so you can jump to the ones that interest you most:

Absent Values
Error Handling
Collections
Concurrency and Parallelism
Polymorphism

Or you can skip straight to my conclusion, which boils down to this:

Despite the strong possibility that this opinion will subject me to ritual humiliation on social media, I would use Scala (or a similarly featured language like Kotlin) for microservices or bigger but Go both to replace any bash or Python scripts that are part of my continuous delivery pipeline and to create lambda functions, which are supposed to be lightweight, fast, and focused.

With that, let me introduce you to Scala and Go.

Scala

Scala is an object-oriented (OO) and functional programming language built for the JVM (though Scala Native is in the works). It emerged from academia at the EPFL in Switzerland from the mind of Academic Director Martin Odersky, who sought to prove that the two paradigms--OO for representing your domain and functional for its mathematical guarantees for working programs--can blend seamlessly to yield the best of both worlds. Scala also has an exquisite static type system that can provide a powerful safety net.

Operationally, as you might expect from a language borne from academia, Scala tooling can be problematic and compilation can be slow--particularly if you are not yet using Scala 3, which only recently emerged and is very slowly percolating through the ecosystem (Remember the Python 2 to Python 3 transition?). But type inference, a vast standard library, and the time-tested reliability of the JVM make you very productive once you get the hang of them. Performance varies with the JVM you're running, but regardless you do have to contend with the size of compiled objects and the latency of garbage collection at runtime. When you want to experiment, you can skip the ceremony of writing a class or test and instead use a command-line REPL, an online REPL called Scastie you can share, or an outstanding third-party command-line REPL called Ammonite. Dependency management is achieved with SBT typically but also more general JVM build tools like Gradle and Maven.

Scala became really popular with the advent of "Big Data" because functional programming lends itself so naturally to analytics, and the learning curve for
modern LISPs like Haskell and Clojure is too high for too many. Apache Spark is built in Scala, and when it got big, Scala got big. Since then Scala has also become a popular language for other domains including reactive web applications and microservices with Play Framework and Akka and even the front end with Scala.js.

Go

Go had an pragmatic mission from the start, so it was built with simplicity, minimalism, and performance in mind. Robert Griesemer, Rob Pike, and Ken Thompson created Go at Google as an alternative to C++ built for modern machines. Go is compiled to native machine code; no virtual machine. Go is statically typed like Scala but is imperative and procedural. It's not a functional language per se, but functions are first-class. Where it
truly shines is in its concurrency primitives: channels and goroutines. Together, they enable you to leverage the full power of your multicore machine.

Operationally, Go is really fast to compile and run. It too has a vast standard library for common tasks like REST calls and JSON de/serialization. When you want to experiment, you can use the Go Playground or a scratch file in your IDE, but there is no command-line REPL. Unlike Scala, Go has by design a very lean feature set and simple constructs. This makes it relatively easy to learn. However, you have to add a lot of features yourself that you take for granted in other languages or explore what Go offers as potential workarounds. To get really good at advanced features of Go like concurrency and polymorphism is still a challenge. Official dependency management is nonexistent, and you may find Go's unorthodox project setup based on Git repos takes getting used to.

After Go took off at Google and was released to the public, it got really popular as the language of concurrency, which helped in turn to make it the language of DevOps--particularly in concert with Kubernetes, which also emerged from Google. Go has expanded into other domains as well with the CMS Hugo and the microservices framework Go kit.

Comparing Scala and Go

Let's look at how Scala and Go handle the kinds of real-world problems you will encounter every day.

Disclaimer: The sample code is not meant to showcase necessarily the "best" way to solve these problems--most performant, most elegant, or whatever. Instead it is meant to showcase reasonable, idiomatic solutions and, more importantly, how they reflect the design philosophies of the respective languages. Of course you are welcome to suggest improvements regardless. Also, the Scala code uses some of the revolutionary Scala 3 syntax and constructs.

Absent values

You have to deal with potentially absent values all the time like when database queries for single entities return no hits or when you are maintaining backwards-compatible microservices. Typically, absent values are represented with null. Sir Tony Hoare called his invention of null to represent the absence of a value his "billion-dollar mistake." Handling absent values isn't glamorous, but if you do it poorly, you will suffer significant productivity loses.

Scala

Although null exists in Scala as a JVM language, you should (almost) never interact with it directly. Instead, you should work with Option, a monad designed specifically for this purpose. We describe Option in more detail in our tutorial, but essentially it compels you to account for the potential absence of a value at compile time. This avoids the costly NullPointerException at runtime that has sent thousands of Java developers to therapy.

case class Student(name: String, house: String)

def findStudent(key: Int): Option[Student] = {
  val students = Map(
    1 -> Student("Harry Potter", "Hogwarts"),
    3 -> Student("Draco Malfoy", "Slytherin")
  )
  students.get(key)
}

findStudent(3)
  .map(s => s"The student's name is ${s.name}")
  .getOrElse("Back to Hogwarts")

In this example, the findStudent function returns an Option[Student]. If the Option contains a value, the client can transform it into a String (i.e. Option[Student] => Option[String]) with the map function on Option; otherwise, getOrElse handles the absent value.

Option allows you to safely work with potentially absent values without fear. The compile-time safety makes you very productive. On the other hand, every transformation on the Option--via map, filter, etc.--produces a new value because of the functional programming bias towards immutability. If memory is at a premium, maybe this is a concern.

Go

Go handles potentially absent values through completely different idioms. Functions can return multiple return values. When a value is present, it's idiomatic to return the value and a bool value (named ok by convention) of true; otherwise, it returns the "zero value" of the return type and false. The client uses an imperative if statement to distinguish.

package main

import "fmt"

type Student struct {
    name string
    house string
}

func findStudent(key int) (Student, bool) {
    students := map[int]Student{
       1: Student{name: "Harry Potter", house: "Hogwarts"}, 
       3: Student{name: "Draco Malfoy", house: "Slytherin"}
    }
    student, ok := students[key]
    return student, ok
}

func main() {
    if student, ok := findStudent(3); ok {
        fmt.Printf("The student's name is %v\n", student.name)
    } else {
        fmt.Println("Back to Hogwarts")
    }
}

In this example, the findStudent function returns a Student and a bool. Go allows you to initialize and test conditions in one line, and that's what we see here. If ok is true, we know we got something and act accordingly; if ok is false, we know the value is absent and handle this contingency.

For those uncomfortable with monad composition and higher-order functions, Go provides a very simple alternative in keeping with its mission. On the other hand, Go engineers need the knowledge and discipline to apply these language features and idioms. There is no dedicated type like Option to help. Also, unlike with the composability afforded by monads like Option, if you need to compose multiple potentially absent values, you need to write multiple if conditions; that can feel verbose. The simplicity may well be worth it though.

Error Handling

This is another inglorious task that is critical to good software engineering. As programs become big and complex, proper error handling is critical to diagnose bugs and move builds to production as quickly as possible.

Scala

As you might imagine from a language that prizes on immutability and composability, Scala offers another monad, Try, for error handling. Analogous to Option, it compels you to account for a possible error at compile time rather than the absence of a value. A Try[Double], for example, represents either a Double if all is well or an error (or more precisely an instance of Throwable) otherwise.

import scala.util.{Failure, Success, Try}

def squareRoot(value: Int): Try[Double] = {
  if (value >= 0) {
    Success(Math.sqrt(value))
  } else {
    Failure(new IllegalArgumentException("Value cannot be negative"))
  }
}
val sum = for
  a <- squareRoot(4)
  b <- squareRoot(16)
yield a + b
sum.map(s => s"The result is $s") match {
  case Success(result) => result
  case Failure(t) => t.getMessage
}

In this example, the squareRoot function returns Try[Double], and the client does something a bit more complicated than the prior example. It calls squareRoot twice and uses a for comprehension, which is syntactic sugar for otherwise cumbersome map and flatMap transformations, to take advantage of the composability of monads to sum the results. If both calls work out, then the result is a Try with the sum; if either fails, the error information is passed on. You can do similar with Option too. This is why monad composability in Scala is so cool. The same pattern works on very different types as long as they follow the monad laws.

As before though, keep in mind that you are generating new objects with each transformation. This means you're paying for the protection immutability affords you with memory.

Go

Just as Option and Try in Scala are similar, handling absent values and handling errors in Go is similar. Here again Go takes advantage of multiple return values, but rather than a value accompanied by a bool, idiomatic Go error handling features a value accompanied by an Error (named err by convention). If err has the value of nil, then you can work with the value because it's all good; otherwise, you handle the error and (probably) ignore the zero-value.

package main

import "fmt"
import "math"

func squareRoot(value int) (float64, error) {
    if value >= 0 {
        return math.Sqrt(float64(value)), nil
    } else {
        return 0, fmt.Errorf("invalid input: %v", value)
    }
}

func sumRoots(v1, v2 int) (float64, error) {
    a, err := squareRoot(v1)
    if err != nil {
        return 0, err
    }
    b, err := squareRoot(v2)
    if err != nil {
        return 0, err
    }

    return a + b, nil
}

func main() {
    if sum, err := sumRoots(4, 16); err == nil {
        fmt.Printf("The sum is %v\n", sum)
        return
    } else {
        fmt.Println("Value cannot be negative")
    }
}

In this example, the sumRoots function, which is the client of squareRoot, returns a value and an error. Those are saved from the first call to squareRoot. If there is an error, the function returns immediately; otherwise, it does the same with the second call to squareRoot. If the function makes it to the end, then it returns the sum and a nil error. You will see a similar approach in main with the call to sumRoots.

There are a few interesting things to note. First, once again the code reflects Go's bias toward simplicity. Furthermore, as immutability is not a priority, variable reuse is common in Go. In this case, err is reused to store the error returned from squareRoot. It doesn't happen here, but it isn't unheard of to reuse the value variable as well once its initial value has served its purpose. Finally, we see a common pattern:

Call a function
Test err for nil
If an error is found, return
Call the next function
Test err for nil
If an error is found, return
Lather, rinse, repeat

Absent Scala's function composition, it's a fair bit of boilerplate--particularly when you are calling a lot of functions that may return errors. You can take advantage of language features to make it more elegant like
this pattern utilizing interfaces and pointers from Rob Pike, but this is one of the ways where Go's simplicity is a bit overrated. Building custom abstractions from Go's toolkit requires creativity, and you will have to do that often when you use Go to build mature applications. Still, it is almost certainly easier to master advanced patterns in Go than it is in Scala.

Finally, even though it doesn't appear in this example, defer is a simple but powerful construct in Go that executes a call at the end of a function no matter what happens--error or otherwise. You can use defer to clean things up after an error in the same way you'd use finally in other languages.

Collections

I don't have to tell you that manipulating data from a database, a stream, a REST request, or a host of other sources is a common task in application development. Languages that facilitate seamless transformation and aggregation of collections of data make your life a lot easier.

Scala

Scala has a vast library of collections--both immutable and mutable though immutable is preferred--each of which offers in turn a vast collection of higher-order functions that let you manipulate the collection in numerous ways.

val words = List("Fear", "of", "a", "name", "only", "increases", "fear", "of", "the", "thing", "itself")

words
  .groupBy(_.toLowerCase)
  .mapValues(_.size)

In this example, given a List of words, the code uses groupBy to convert it into a Map of each word (lowercased to normalize them) to a list of occurrences of each word. Finally, those lists are transformed into their sizes, and the result is a Map of each word to its count.

There isn't much to see. This is a credit to Scala's powerful abstractions. Still, it is important to keep in mind that this code performs multiple O(n) traversals and with each one consumes memory to produce an entirely new collection--all but the first and last of which are immediately thrown away.

Go

Go naturally takes a far more lightweight approach to collections. You will essentially only deal with maps and slices, which are array views that enable memory-efficient operations on the backing arrays.

package main

import "fmt"
import "strings"

func main() {
    dictionary := make(map[string]int)
    words := []string{"Fear", "of", "a", "name", "only", "increases", "fear", "of", "the", "thing", "itself"}
    for _, word := range words {
        word = strings.ToLower(word)
        if count, ok := dictionary[word]; ok {
            dictionary[word] = count + 1
        } else {
            dictionary[word] = 1
        }
    }

    fmt.Printf("The counts are %v\n", dictionary)
}

In this example, the code uses make both to create an empty map (with values defaulting to the zero value, in this case literally 0) to hold the word counts and to create a slice containing the words. It simply iterates over the slice and builds the word count using the ok idiom you saw before to check if there is an existing entry for the word in the map.

The code is more verbose than its Scala counterpart, but it is significantly more efficient in time and space. There is a single O(n) traversal with constant-time lookups in the map, and we maintain only two collections the entire time. The efficiency of Go collections and the simplicity of using them are among the best reasons to use Go in a project.

Concurrency and Parallelism

Modern software applications have a lot more to do in a lot less time, so it's important for your code to take advantage of every bit of power your machines have. Modern applications demand modern languages that enable you to leverage every core through abstractions that strike the right balance of power and intuitiveness. Perhaps most importantly, they need to provide mechanisms for handling errors because concurrent/parallel programming is notoriously hard to debug. It's a challenge to reproduce the conditions that generated the bug in the first place.

By the way, as Rob Pike has taught us, concurrency is not parallelism. Long story short, concurrency is about decomposing a problem into its components; each component might then run in parallel depending on available resources. Concurrency manages a lot of things at once while parallelism does a lot of things at once.

Regardless, doing concurrency and parallelism well is a hard problem. Whatever path you choose, you need to understand the nature of your tasks to achieve peak performance. Are they IO- or CPU-intensive? Are they intrinsically parallel?

Scala

As a core language in reactive programming, Scala takes concurrency and parallelism very seriously. It offers Future as its core primitive to facilitate concurrency and parallelism--in concert with an ExecutionContext, which is basically a thread pool. Future abstracts the threads away from you, which is nice, but its association with ExecutionContext can lead to some complexity in understanding how they work. This is why notable third-party libraries offer their own concurrency primitives. Still, Future at least approximates a monad, and that means we can mostly adhere to the familiar patterns we saw with Option and Try.

At a low level, each asynchronous call delegates to a different thread. This helps scale your application, but it's also a heavyweight operation as the operating system needs to schedule threads against physical processors and manage expensive context switching. The result is that for some problems parallelism with Future in Scala may potentially consume a lot of resources for merely a modest increase in performance--or even slow you down. When performance is a major concern, you need to configure your ExecutionContext smartly according to the capability of your machine(s) and the nature of your tasks.

Bottom line? Reactive programming doesn't necessarily make your applications faster (except maybe in those cases where you can do expensive work in parallel), but it usually allows them to be more resilient. Reactive applications scale under load with limited threads and memory especially when you have latency from inconsistent network IO like database and REST calls.

import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import play.api.libs.ws._
import play.api.libs.ws.ahc._

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits._
import play.api.libs.json._

implicit val system = ActorSystem()
system.registerOnTermination {
  System.exit(0)
}
implicit val materializer = ActorMaterializer()
val ws = StandaloneAhcWSClient()

def getUuid: Future[String] = {
  ws.url("https://httpbin.org/uuid")
    .withHttpHeaders("Accept" -> "application/json")
    .get
    .map { response =>
      (response.body[JsValue] \ "uuid").as[String]
    }
}
val uuid1: Future[String] = getUuid
val uuid2: Future[String] = getUuid
val uuids = for
  u1 <- uuid1
  u2 <- uuid2
yield (u1, u2)
uuids
  .foreach {
    case (u1, u2) => println(s"UUIDs: $u1, $u2")
  }
  .andThen { case _ => ws.close() }
  .andThen { case _ => system.terminate() }
  .recover {
    case t: Throwable => println(s"There was a problem: $t")
  }

In this example, the code uses Play WS Standalone as a REST client to fetch JSON containing a UUID. Play WS has an asynchronous, non-blocking API based on Future, so you need to provide an ExecutionContext via Akka. That's all the boilerplate at the beginning of this example. Sometimes it will be done for you as when you use Play WS in the context of Play Framework. Nonetheless, you should be aware it has to happen somewhere.

In the getUuid function, the get call to Play WS makes an asynchronous, non-blocking HTTP GET request and returns a Future containing the response. You need to be careful and make sure you only work directly with the Future itself lest you lose the eventual result--the response or an error. Otherwise anything you do next outside the realm of the asynchronous call will execute after the request is dispatched but completely independently from when the response returns. That's a common mistake for engineers new to Future. This is why everything that happens after the call in getUuid is dispatched is a method call on the Future object that comes back.

Like a typical monad, Future offers map to enable clients to transform results, and getUuid does exactly that by parsing the JSON response into the UUID string and returning a Future[String]. If the REST call returned an error, it remains preserved in the Future. The client of getUuid requires two calls to complete before moving forward, so it calls the function twice so the independent fetches can run in parallel and composes them as we saw with Try via for comprehension to produce a Future[Tuple2[String, String]]. Finally, the code calls map again to transform the tuple of strings into a printed result. If there is an error any step of the way, we handle it with recover.

Future is a really nice concurrency primitive that offers the convenience of a monad, but it has its pitfalls. As mentioned, you need to supply a finely tuned ExecutionContext, and everything you do once you make an asynchronous call better happen in the context of the Future that returns. Otherwise, you will find very confusing results like silent failures. The amorphous referential transparency of Future can confuse you too. If the code made its calls to getUuid inside the for comprehension, they would have been sequential and not parallel. The result is likely the same, which feels referentially transparent, but the fact where you make the call impacts the desired parallelism definitely doesn't.

Future also consumes a lot of system resources because it transacts in full threads that have to managed by the operating system. It's important to profile your application to understand what's going on because there will almost be certainly occasions where things don't perform as you expect. You will need to diagnose if it is a code or resource problem.

Go

Concurrency and parallelism lie at the heart of Go's mission as well, but of course it takes a totally different approach with primitives called goroutines and channels. The philosophy here is to break down your tasks into independent functions, and spin them off into goroutines. The key thing to remember is goroutines are not threads; they are lightweight pieces of memory tracking thread usage. As a result, you can spin off literally hundreds of thousands of goroutines and use only a little memory on the stack while the Go scheduler, not you, uses clever algorithms to manage them in concert with the operating system and its actual threads.

Goroutines use a paradigm called communicating sequential processes (CSP) developed by Sir Tony Hoare, who despite being the null guy is a pillar of computer science. CSP is a message-passing model. Goroutines pass their data over channels rather than synchronize data, which slows things down. If you are familiar with messaging patterns like Gregor Hohpe's Enterprise Integration Patterns, then you understand the power of this approach to manipulate message data as needed to produce the results you want--and at scale thanks to Go.

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
    "strings"
)

type Result struct {
    uuid string
    err error
}

func getUuid(rc chan<- Result) {
    r, err := http.Get("https://httpbin.org/uuid")
    if err != nil {
        rc <- Result{"", err}
        return
    }
    response := make(map[string]interface{})
    err = json.NewDecoder(r.Body).Decode(&response)
    if err != nil {
        rc <- Result{"", err}
        return
    }
    rc <- Result{response["uuid"].(string), nil}
}

func main() {
    n := 2
    rc := make(chan Result, n)
    uuids := make([]string, 0, n)
    for i := 0; i < n; i++ {
        go getUuid(rc)
    }
    for i := 0; i < n; i++ {
        r := <-rc
        if r.err != nil {
            fmt.Printf("There was a problem: %v\n", r.err)
            return
        } else {
            uuids = append(uuids, r.uuid)
        }
    }
    fmt.Printf("UUIDs: %v\n", strings.Join(uuids, ", "))
}

In this example, the code defines a type called Result containing a UUID and an error to hold the result of a concurrent computation. Only one of those should be populated with a meaningful value, but there is no simple way to enforce that. Meanwhile, the main function creates a buffered channel for communicating Result values and spins off two goroutines for two concurrent getUuid calls, which take the channel as a write-only parameter. That is enforceable.

The getUuid function makes the REST call and unmarshals the JSON response. If an error occurs at either step, the code creates a Result with the respective error (and the zero value, the empty string, for the UUID) and publishes it to the channel. If all goes well, the code creates a Result with the UUID (and the zero value, nil, for the error) and publishes it to the channel instead. The main function knows in this case how many Results to expect from the channel and consumes the right amount of data from the channel. It bails at the first error it finds, or it accumulates the data into a slice of UUIDs.

Goroutines are lightweight and flexible--and straightforward if you have a good grasp on messaging. Still, there are subtleties that can make them more complicated than we might expect from Go. If you don't understand the difference between buffered and unbuffered channels, you may find curious results. Error handling is also tricky because patterns aren't obvious. In this case we encapsulated success and error cases in a single struct, but some advise using dedicated error channels. This example is also as simple as it gets: You have one channel, and you know how many computations will transpire. In more complicated cases you will need to utilize other Go functionality from the sync package like Mutex, WaitGroup, Pool. You might even need the experimental ErrGroup. Finally, you need to be very careful with pointers and mutability generally as always in concurrent programming. They save memory but you need to govern access carefully.

Concurrency and parallelism are always hard. You have to decide which language offers primitives that comport with your mental model of how things should work.

Polymorphism

You know polymorphism. Far beyond trite Animal-Dog-Cat examples, the business value of polymorphism is to leverage abstractions to limit changes to your code even as the functionality of your application grows. By defining new behavior leveraged through old abstractions, you can build software efficiently, and you don't have to work weekends when your client demands new features immediately.

Scala

As an OO language, Scala offers the familiar polymorphism that developers in Java, Ruby, and similar languages have loved for years, but because it is a functional language with a rich type system, it also offers
typeclass polymorphism, which enables completely unrelated types to exhibit polymorphic behavior. You can think of it as functional programming's take on the Open-Closed Principle from OO. Perhaps most striking of all
in comparison to Go, Scala offers parametric polymorphism--what the kids call "generics." When Java 5 introduced generics, it was revolutionary, and Scala benefits as well.

// Runtime polymorphism

trait Closeable:
  def name: String
  def close: String

case class Connection(override val name: String, database: String) extends Closeable:
  override def close: String = s"Closing connection $name"


case class File(override val name: String, opener: String) extends Closeable:
  override def close: String = s"Closing file $name"


def showClosing(c: Closeable): String = c.close

val w = Connection("MyConnection", "PostgreSQL")
val f = File("file.txt", "TextEdit")
showClosing(w)
showClosing(f)


// Parameteric polymorphism (generics) with List[T]

val numbers = List(1, 2, 3)
val firstNumber: Int = numbers.head

val strings = List("1", "2", "3")
val firstString: String = strings.head

def use[T <: File](t: T) = s"Using ${t.name}"

use(f)


// Typeclass polymorphism

case class Complex(real: Double, imaginary: Double)

object Complex:
  given Ordering[Complex] with
    def compare(a: Complex, b: Complex): Int = a.real.compare(b.real)


List(Complex(3, 5), Complex(10, 2), Complex(1, 6)).sorted

In this example, we really see three examples of polymorphism in Scala. The first is the kind of straightforward runtime polymorphism familiar to Java developers--except with Scala traits rather than Java interfaces. Connection and File are marked as instances of Closeable, and the showClosing function accepts a Closeable parameter. Therefore either Connection or File is suitable to pass to showClosing and the result of the function is dynamically and polymorphically resolved.

The second example of polymorphism is also familiar to Java developers. It's generics. In Scala, you never have just List; you have List[T], where T is a type parameter indicating the type of item in the List. Scala's type inference deduces that numbers is an instance of List[Int]; strings is an instance of List[String]. The head method returns the first element of a List, which is a T. T is Int with numbers and String with strings. Finally, just for demonstration purposes, the code defines a function use that's not only type parameterized but type bounded. It only accepts a type that's File or any subclass of File. If you try passing a Connection to it, the code won't compile.

The last example is what I consider the coolest and most powerful form of polymorphism in Scala--typeclass polymorphism. List has a sorted method as you'd hope, but it requires an implicit ordering. In other words, you have to tell List[T] how to sort its elements by providing a function that lifts T into an Ordering[T]. This makes perfect sense, and it is enforced by Scala's type system. The code defines a type called Complex and a way to lift Complex to Ordering[Complex] that defines how instances of Complex should be sorted. Without this, List[Complex] wouldn't know how to sort its contents, and that last line wouldn't compile. This means you can make any T sortable by defining an Ordering[T] typeclass. More broadly, it means you can use typeclasses to add polymorphic behavior to completely unrelated types, which includes types you don't control like legacy types and/or types found in imported dependencies.

Polymorphism in Scala is powerful and flexible because of its sophisticated type system. It allows you not only to extend functionality in clever ways but also to constrain the solution space. In other words, you can limit the number of ways a problem can be solved, which makes it harder to write bugs.

Go

Go is not an OO language. Structs have no capacity for inheritance by design--only composition via "embedding". Go is not quite functional either. However, polymorphic behavior is not only possible but a fundamental part of the power of Go via structural typing. You can take advantage by doing two things. First, you write interfaces, which as usual define the method signatures for a set of API calls. You can also compose interfaces via embedding. Second, you can endow any type--an existing Go type like float64 or your own custom structs--with behavior by defining functions and assigning them to the type. When you do this, the type is called a "receiver", and if the receiver has been assigned all the functions associated with a given interface, it is an implicit instance of that interface. You can then pass the type to any function expecting an instance of that interface, and it's resolved at compile time, which makes you more productive in stark contrast to the runtime resolution of duck typing in dynamic languages like Python.

package main

import (
    "fmt"
)

type Named struct {
    name string
}

type Connection struct {
    Named
    database string
}

type Closeable interface {
    close() string
}

func (c Connection) close() string {
    return fmt.Sprintf("Closing connection %v", c.name)
}

type File struct {
    Named
    opener string
}

func (f File) close() string {
    return fmt.Sprintf("Closing %v", f.name)
}

func showClosing(c Closeable) string {
    return c.close()
}

func main() {
    connection := Connection{Named{name: "MyConnection"}, "PostgreSQL"}
    file := File{Named{name: "file.txt"}, "TextEdit"}
    fmt.Println(showClosing(connection))
    fmt.Println(showClosing(file))
}

In this example, interface Named is embedded in Connection and File. Note that this does not denote any relationship among them, but there is a tight coupling similar to inheritance in that any changes to Named are reflected wherever it is embedded. Interface Closeable is defined with a single function close with no parameters and returning a string. Any type with an identical function is resolved at compile time as an instance of Closeable, and lucky for us, Connection and File qualify as they are both receivers of a close function with the right signature. As a result, showClosing works just fine when called on both kinds of structs.

That's the extent of Go's polymorphism. Clearly it is not remotely as extensive as Scala's, but it promotes two important values--composition over inheritance and abstraction over implementation. Engineers coming from traditional OO backgrounds may find Go's polymorphism takes a little getting used to, but with some creativity you will find it
quite powerful. There is no question, however, that the absence of generics
can be quite jarring for more complex applications. It's such a powerful and pervasive idiom that even front end languages like TypeScript and Elm have it. As it happens, there has been such demand for generics from the Go community that the maintainers have begun considering it. As the debate rages on whether the benefits outweigh the costs--potentially the speed and simplicity fundamental to Go's mission--just recognize you won't have the benefit of generics for a while.

How Do You Decide?

Scala and Go are two great languages with fundamentally different philosophies that offer distinct advantages and disadvantages. I've tried to lay those out as simply as I can so you can extrapolate which might be best suited to your situation.

Having worked on complex applications with both languages, I can tell you Scala takes longer to grasp, and you will have a harder time finding Scala engineers--especially in the United States and especially if you require onsite work. But if you have senior staff who can mentor novices and who can build abstractions that utilize functional programming's strengths
and the exquisite type system to constrain wayward novices, you will find the team growing very productive very quickly. Day-to-day tasks like compilation and continuous delivery are slower, and you will often find yourself exploring Scala's rich open-source community to enhance development.

Meanwhile, anyone can learn Go. The constructs are simple and lightweight, and compiling and executing are just so fast. It's amazing. Mastering the more advanced concepts of Go, however, demands effort. You will also find yourself reinventing the wheel from other languages often--like writing your own filter function, which is easy enough but is more plumbing than directly related to your business domain--and building creative workarounds for the limitations of Go by exploiting the powerful features Go does offer. Dependency management and error handling could be better, and the absence of generics can be rather painful when dealing with unknown schemas like unmarshaling dynamic JSON.

So it all depends on if the priorities of your application and project align with the priorities of the language and ecosystem you choose. Despite the strong possibility that this opinion will subject me to ritual humiliation on social media, I would use Scala (or a similarly featured language like Kotlin for microservices or bigger but Go both to replace any bash or Python scripts
that are part of my continuous delivery pipeline and
to create lambda functions, which are supposed to be lightweight, fast, and focused. This means you may not necessarily have to choose because nontrivial cloud native architectures will often blend both microservices and lambdas, and you should always have a continuous delivery pipeline.

I hope this helps. If you read the whole thing, you deserve a nap.

Code Coverage Is Killing You

Neil Chaudhuri — Sun, 15 Aug 2021 20:48:27 +0000

If you are a software engineer or run software projects, code coverage is probably very important to you. It's intuitive. Of course more tests produce better software! It's easy to calculate. Tools, automation, and stunning charts to impress the people who pay for the occasional pizza are all readily available.

The problem is code coverage is killing you.

Don't get me wrong. You deserve credit for your agile commitment to quality and your investment in continuous integration and continuous delivery. But why does just about everything out there say 100% code coverage is [at best unrealistic and at worst dangerous? How can achieving a perfect score on a great metric be a bad thing? But OK. What's the optimal number then? There are a lot of heuristics, but no one really knows.

All of this uncertainty over an apparent no-brainer should give you pause, but it is important to understand the main reasons why code coverage is so flawed.

Code coverage assumes all your code is equally vulnerable

Let's say your code includes a credit card validator. There
is an infinite number of ways a string can fail credit card validation, but you only need a few tests to achieve 100% code coverage for it. Meanwhile other code may have far fewer failure scenarios. Ideally, that credit card validator should have hundreds of tests associated with it while other code has far fewer. Code coverage treats them all the same and only credits you for a small fraction of the tests necessary for your most vulnerable code, so you probably won't write any more than you have to. I believe that's what the kids call "perverse incentives."

Code coverage assumes all your code is equally valuable

Let's say your application also offers payment via money order. Almost no one will use that option; the vast majority of customers will pay by credit card. Code coverage doesn't know the difference, so it considers credit card tests and money order tests equally important. That's bad. A bug with credit card payments will cost you orders of magnitude more profit than a bug with money orders, but code coverage will never account for that.

Code coverage demands ceremony that may not be necessary

Let's not lose sight of the goal, which is to gain as much
confidence as possible that your code works. Certainly that can come from tests, but it does not have to. For example, if you are building a component library in React that will be consumed by application developers, it could very well be Storybook is all you need. After all, a button in your component library will have no functionality on its own. Its functionality will come from the onClick event handler passed as a prop by the library consumer. All you care about is that clicking the button does something. It's a waste of time to write a test for something so trivial just to check a coverage box when Storybook gives you that for basically free. Of course, if you insist, you can also
use Storybook stories as fixtures for your tests.

Code coverage tells you how much but not how well

This could be the worst of all. You are investing a lot of budget and schedule in building a test suite that should raise the quality of your software, lower costs, and improve customer satisfaction. But while code coverage tells you how much you tested, it doesn't tell you how well you tested. You can write really poor tests (e.g. tests that don't offer any challenging scenarios, tests that are flaky or slow, etc.) that yield 100% coverage. In the end, those tests might actually be counterproductive because the investment could have gone elsewhere and yielded some value.

Frankly, none of these concerns are debatable. The only real argument for code coverage is that for all its flaws it is still the best we can do. Obviously I cannot speak to your specific project, but my extensive experience suggests that your best case scenario--if you have a team of talented, disciplined engineers who have faced no deadline pressures to get features out fast--is that your dedication to high code coverage has improved quality but not nearly enough given how much you've spent trying. Instead, what's most likely is all your spending has produced mostly useless tests that exist to check a box rather than to improve quality, and your code is only barely better than with no tests at all.

The good news is we can do a lot better for a lot cheaper.

Percentage Metrics

Experience has shown me there are a lot better metrics than code coverage. Here is a good list of percentage metrics inspired by Kostis Kapelonis.

Percentage of Bugs Reproduced By Tests (Target: 100%). This is the best metric. Every bug reported by testers or users should have at least one test associated with it.

Percentage of Tests That Change (Target: 0%). Too often tests are coupled to implementation, so updates to your implementation details--a new technology, updated algorithm, etc.--lead to laborious updates to tests. That should stop. Shifting away from conventional, scenario-based testing to
property-based testing where possible can help.

Percentage of Consistent Tests (Target: 100%). Have you ever seen the same tests pass some days and fail others? That's not consistent, and engineers typically respond by disabling or commenting out offending tests. Instead, rewrite these tests to be deterministic unit tests, or delete them entirely.

Percentage of Developers Writing Tests (Target: 100%). Everyone who writes code should write tests, but it is important not to take this in a punitive direction. The purpose is not to uncover slackers; it's to grow a culture where all developers recognize test code deserves the same care as production code and to expose the entire team to the entire codebase to promote collective code ownership. If command-and-control types do try to use this metric to identify what they consider "weak performers," then get rid of it. It does more harm than good.

Trending Metrics

There are also trends to watch.

Low Code Quality in Tests. Tests are code. Your test code should be a first-class citizen subject to the same quality checks as production code--limited duplication, reusable functions, design patterns where useful, etc.

Time to Write Tests. Maybe the quality of your tests is too low. Maybe the design is poor with too many dependencies to mock. Maybe it's hard to generate test data or scenarios. Taking too long to write tests will manifest in diminishing velocity and more bugs. Better design and again
property-based testing can help.

Time to Run Tests. One of the core tenets of agile development is rapid feedback, which is impossible if your tests take forever. This might be controversial, but I would recommend favoring unit tests and functional tests over integration tests. Unit tests are fast (when written properly). Functional tests give you the most accurate view on quality, and modern tools like Cypress can overcome the slowness and flakiness of older tools like Selenium. You can also get a boost from your tooling.

Keep these trend lines as low and as level as possible.

The best part about all of these metrics is that they are easily derived from your engineering tools. Nothing special is required of you. Of course you can always take the initiative to do some clever things specific to your domain like identifying particularly vulnerable and/or valuable parts of the codebase and ensuring there is a high level of coverage for that specific region of surface area.

In the end you should be able to look at every test in your code base and recognize how it gives you confidence that your code will work in production and that no one will be working weekends. The siren song of code coverage is intoxicating, but you can do so much better.

Dark Mode in Next.js Using Tailwind CSS and React Hooks

Neil Chaudhuri — Tue, 03 Aug 2021 01:18:50 +0000

It's quite possible that while waiting for the ads on Hulu to end you stumbled upon the option to set your phone's theme to Dark Mode. Dark Mode is becoming a staple of user interfaces on the web and mobile devices for several reasons--primarily to ease the strain on your eyes and to reduce battery consumption.

At Vidya we pride ourselves on embracing emerging technologies and helping our clients leverage them to realize their potential. When it came time to give our website a fresh new look, we figured adding a toggle-able Dark Mode option would be consistent with that mission.

The site is built in TypeScript with React, the most popular JavaScript library in the world, using Next.js, one of the most popular React frameworks in the world and the building block for full-stack "meta" frameworks like RedwoodJS and Blitz. The user interface itself is crafted with the ever popular Tailwind CSS, a powerful "utility-first" library that lets you compose your styles into higher-level abstractions that you apply across your user interface
to give a consistent look and feel.

If you would like to implement Dark Mode on a Next.js site using TailwindCSS, let me show you how. It involves three key pieces:

Tailwind's dark class
The Script tag that we got in Next.js 11
Understanding, like really understanding, React's useEffect hook

Activating Tailwind's Dark Mode Support

Tailwind CSS offers two ways to set Dark Mode. If you are content to default to system settings, then all you need to do is confirm your tailwind.config.js file has the media setting, which uses the prefers-color-scheme CSS media feature:

// tailwind.config.js
module.exports = {
  darkMode: 'media',
}

But since we want more control to let Vidya users decide which look they prefer, we need the class setting instead:

// tailwind.config.js
module.exports = {
  darkMode: 'class',
}

Now you need to handle variants like the TVA in Loki. Variants in Tailwind define the ways in which you want to apply different styles. For example, if we want to set a red background on a link hover, we apply the hover variant on the bg plugin: <a className="hover:bg-red">.

As an aside, the CSS equivalent would be this for our shade of red:

a:hover {
  background-color: #9C4D61;
}

We will do similar to apply dark variants of our branding scheme throughout our interface. For example, here is a simplified version of our contact-us class composing numerous Tailwind utilities in Next.js's globals.css file:

.contact-us {
        @apply dark:text-red dark:hover:text-blue bg-red dark:bg-red-light hover:bg-blue-dark dark:hover:bg-blue-light;
}

Note that you always put dark first when you have multiple variants like dark:hover:bg-blue-light.

This is where you will spend most of your time. Mostly because you want to put together a Dark Mode color palette that is usable and accessible and consistent with your branding and because you want to be thorough in applying it throughout the site.

Just remember to extract components as we did above to keep things maintainable, consistent, and organized.

Because we are relying on the Tailwind class setting for Dark Mode, we need to figure out a way to hook the dark class onto the root element of each page like this:

<html lang="en" class="dark">
...
</html>

And we need to be able to do it on demand. This is where our code comes into play.

The Script Tag

If you've built a website with a lot of client side business functionality, GDPR or other consent management, Google Analytics, social media, or ads, you already know that managing JavaScript execution has always been awkward. Where do you put this script on the page relative to that one? Do you put this script at the top of the head element or at the bottom of the body element? It's actually easier figuring out where to seat everyone at your wedding.

In v11.0.0, Next.js introduced the Script tag, and it makes all this a lot better. You can put the Script tag anywhere, and you apply one of three strategies to let Next.js know when it should execute.

Before we specify which strategy should apply here, keep in mind our goal: to assess the user's Dark Mode preference and apply it immediately. For this script to work, it must execute before the browser paints the page, so it has to block interactivity. This contradicts everything you've ever read about script optimization. Conventional guidance dictates scripts should run in an asynchronous, parallel fashion in order to maximize Web Vitals and get the user up and running as soon as possible. That general guidance is accurate, but we need to make an exception for this particular script. Still, it must execute very quickly, or we will lose customers.

Our strategy for implementing Dark Mode will factor in potential user preferences specific to the Vidya website set in localStorage,a key-value store available in modern browsers, and/or system settings that the browser will inform us with prefers-color-scheme. The algorithm goes like this:

If the user previously visited the Vidya website and indicated a preference for Dark Mode OR if there is no preference established and system settings are set for Dark Mode, then activate Dark Mode by attaching the dark class attribute to the root. Otherwise, apply Light Mode by removing any dark class attribute.

Here is the darkMode.js script that does exactly that:

if (localStorage.getItem('vidyaDarkMode') === 'true' || (!('vidyaDarkMode' in localStorage) && window.matchMedia('(prefers-color-scheme: dark)').matches)) {
    document.documentElement.classList.add('dark')
} else {
    document.documentElement.classList.remove('dark')
}

That's a straightforward conditional, which might even short-circuit, and DOM manipulation. That should be fast. Phew!

And here is how we execute it before browser paint with Next.js's Script tag inside our _app.tsx:

import Script from "next/script";
// ...
<Script strategy="beforeInteractive" src="/scripts/darkMode.js"/>

The beforeInteractive strategy is the key. This tells Next.js to block everything until the script is finished. Again, you need to use this strategy very carefully, but it's necessary and proper in this instance.

So thanks to Tailwind CSS and Next.js, we can successfully apply Dark Mode based on user preferences one way or another when the Vidya website loads. The last step is to give the user a chance to switch modes and to save that preference to localStorage.

With Great Effects Come Great Responsibility

When Facebook revolutionized React with Hooks, it was a game changer, but even now, years later, they can be confusing. Let's see how we can use useState and useEffect to complete our Dark Mode solution.

The work we did with Tailwind CSS and the Script tag presents our user interface exactly as it should look from what we know so far, but React needs to manage that preference to change it as the user dictates. There are two steps:

React needs to be made aware of the initial Dark Mode preference and keep an eye on it.
If the user changes that preference, React needs to add or remove the dark class from the root and persist the choice in localStorage accordingly.

These are two different effects. We will localize them where they matter most, the ThemeButton the user clicks to switch modes.

Before we get into those, lets prepare to maintain state:

const [darkMode, setDarkMode] = useState<boolean | undefined>(undefined)

Although we really want darkMode to be true or false, we need to initialize it with undefined because we don't know what it is until the first effect runs.

Here it is:

useEffect(() => {
        setDarkMode(document.documentElement.classList.contains("dark"))
}, [])

It's simple but deceptively so. It's really very very sneaky.

Note the empty dependency array. Many React developers, especially the other old timers who remember the awkwardness of handling effects in component lifecycle events, think of this as the equivalent of the initial set up we did in componentDidMount. That way of thinking can work for you, but it's imprecise and I would say counterproductive to understanding how React works.

The purpose of useEffect is to synchronize UI with the state represented in the dependency array. When that state changes, UI changes. However, the absence of dependencies means that you want to synchronize your UI with the absence of state, and state just happens to be absent when a component first mounts. So yeah, it works out the same as that componentDidMount analogy, but they're really two different things.

This is why math teachers make you show your work.

As a result, this first useEffect call runs when state is absent as the component initially mounts, and the current darkMode value is saved to state. We can deduce the value from the root element because of the code we wrote earlier using the Next.js Script tag, which we know has already executed because we used the beforeInteractive strategy.

See how it all fits together?

Finally, there is the second hook that triggers and records a change to the theme when the user clicks the button:

useEffect(() => {
        if (darkMode) {
            window.document.documentElement.classList.add('dark')
            localStorage.setItem("vidyaDarkMode", "true")
        } else {
            window.document.documentElement.classList.remove('dark')
            localStorage.setItem("vidyaDarkMode", "false")
        }
}, [darkMode])

const onClick = () => {
        setDarkMode(!darkMode)
}

This is a more straightforward implementation of useEffect. The darkMode state value is in the dependency array of the effect, so when the user clicks the ThemeButton and toggles the value with setDarkMode, two effects execute. The code modifies the root element by adding or removing the dark class as needed and persists the setting to localStorage so our Script from before will pick it up again when the user returns to the Vidya website.

Let's wrap up by putting together all the relevant Dark Mode logic in ThemeButton :

export const ThemeButton = (p: ThemeButtonProps) => {
    const [darkMode, setDarkMode] = useState<boolean | undefined>(undefined)
    useEffect(() => {
        setDarkMode(document.documentElement.classList.contains("dark"))
    }, [])
    useEffect(() => {
        if (darkMode) {
            window.document.documentElement.classList.add('dark')
            localStorage.setItem("vidyaDarkMode", "true")
        } else {
            window.document.documentElement.classList.remove('dark')
            localStorage.setItem("vidyaDarkMode", "false")
        }
    }, [darkMode])
    const onClick = () => {
        setDarkMode(!darkMode)
    }

    return ( {/* ThemeButton UI goes here */} )
}

So that's it. I hope it's clear how the different components of our solution complement one another to bring Dark Mode to the Vidya website, but this is just one way of doing it. I can't wait to see how you apply the lessons learned here to build great Dark Mode experiences for your audience as well. If you come up with a better way of doing it, please let us know.