DEV Community: Rocky Warren

AWS SQS Best Practices

Rocky Warren — Sat, 20 May 2023 20:44:55 +0000

Context

AWS Simple Queue Service (SQS) provides fully managed message queuing for microservices, distributed systems, and serverless applications.

Best practices

Consume messages in a separate thread or process

Node is single threaded; it can only operate on one task at a time. Creating a separate process creates isolation, preventing API availability issues if, for example, consumers spike CPU or memory usage.

Consumers should be idempotent

Both EventBridge and SQS provide “at-least once” message delivery. That means duplicate messages WILL, at some point, happen. Consumers must handle this gracefully. A simple way to avoid duplicate writes is to use producer-provided IDs that consumers use to deduplicate messages.

Process messages quickly or set the proper configuration

From AWS’s SQS Best Practices:

Setting the visibility timeout depends on how long it takes your application to process and delete a message. For example, if your application requires 10 seconds to process a message and you set the visibility timeout to 15 minutes, you must wait for a relatively long time to attempt to process the message again if the previous processing attempt fails. Alternatively, if your application requires 10 seconds to process a message but you set the visibility timeout to only 2 seconds, a duplicate message is received by another consumer while the original consumer is still working on the message.

If using the sqs-consumer library and you know how long it takes to process a message batch, configure the visibility timeout. Otherwise, configure the heartbeat interval.

Configure a dead-letter queue (DLQ) with maximum message retention (14 days)

Details. If a consumer fails to process a message N number of times (configured via SQS’s maxReceiveCount), the message is sent to the DLQ. This prevents “poison pill” messages from continuously failing, which negatively affects consumer throughput.

Avoid automatically consuming DLQs

If you consume DLQ messages and delete them, they’re gone forever. Instead, set the maximum 14 day message retention. This provides a safe place for messages to sit while you root cause the issue causing messages to end up there. Once a fix is deployed, you can then consume DLQ messages to bring your service’s data up-to-date.

Handle partial batch responses

If your handler returns without error, sqs-consumer deletes the messages from the queue. Throwing an error, however, fails an entire batch. To delete them instead, return a list of successful messages from the handler. For Lambdas, see this.

For Lambda, set visibility timeout to 6x the function timeout and redrive policy to at least 5

To allow your function time to process each batch of records, set the source queue's visibility timeout to at least six times the timeout that you configure on your function. The extra time allows for Lambda to retry if your function is throttled while processing a previous batch.

To give messages a better chance to be processed before sending them to the dead-letter queue, set the maxReceiveCount on the source queue's redrive policy to at least 5.

Source

References

PrivacyProtect: Securely Share Passwords and Store Sensitive Files in Convenient Locations

Rocky Warren — Mon, 30 Jan 2023 00:00:00 +0000

My dad sent me a tax document as an email attachment.

Email is insecure. So are most messaging services.

Now Google (I use Gmail) and Microsoft (he uses Hotmail/Outlook) have my dad's juicy personal information stored; who knows where and for how long.

Why is sharing sensitive information hard?

The top three Google results for "send files securely" are:

Dropbox
A service called "Send Files Securely"
A Wired article listing seven "best bets"

Dropbox and Send Files Securely aren't end-to-end ( E2E ) encrypted. Neither are five of the seven on Wired's list. Of the remaining two, Firefox Send doesn't exist anymore, and iCloud offers it, but only on the newest devices on an opt-in basis. It also disables access to iCloud.com by default and comes with other caveats that may discourage casual users.

E2E encryption is what we want for sensitive data. Encryption "in transit," "at rest," and/or "in storage" is great, but E2E encryption is better. Without it, there are periods when your data is readable, and many companies take this opportunity to do just that. This is how handy features like in-document search work.

But sometimes, we don't want Apple, Google, and the others snooping around. We have files or images for one person's eyes only. We want our private data left alone.

Most password managers let you do this. But I can't convince my dad (and other family members) to use one. Requiring an account to send files or one-off secrets is a small but genuine hurdle keeping people from doing the right thing.

So, when I heard about Portable Secret, I smiled. A dead simple way to share and store passwords, files, and images. However, the creator, Marco, has little time for upkeep, and the security landscape changes fast.

Enter PrivacyProtect. Share passwords and sensitive files over email or store them in insecure locations like cloud drives using nothing more than desktop or mobile web browsers like Chrome and Safari.

PrivacyProtect: share and store passwords and sensitive files with end-to-end encryption.

No special software. No need to create an account. It's free, open-source, keeps your private data a secret, and leaves you alone.

It's so easy, my dad can use it.

Give it a try

Enter your secret message or upload a file you want to protect.
Create a password.
Create an optional password hint to either help you remember the password or help the recipient guess it.
Click Conceal and download secret to generate your protected HTML file and download it. Depending on your device's speed, it can take a few seconds.

And just like that, your secret is protected!

Assuming you chose a strong password , the downloaded file is safe to share over insecure channels like email or messaging services. To view the secret, the recipient (which may be you) will:

Save the HTML file to their device.
Open it in a web browser.
Enter the password. If the recipient cannot guess the password based on the hint, give them a call and let them know.
Click Reveal secret.

Here are examples of a PrivacyProtected message and image file. Try "dog" for the password.

You can also safely store the HTML file in insecure locations like cloud drives (Dropbox, iCloud, Google Drive), host it like the examples above, or keep it on your computer, phone, or USB. Even if you lose your USB drive, the secret is unreadable by anyone that finds it.

Get PrivacyProtect updates

Get notified of new features. Unsubscribe at any time.

Join

Use cases

PrivacyProtect is a complement, not a replacement, for password managers. We've discussed a few use cases above. Here are other ways to use it.

Have you sent sensitive tax documents to your CPA or bought or sold a house that required sending countless PDFs over insecure email? Use PrivacyProtect to conceal them beforehand!
Do you have a crypto wallet? Store your secret recovery phrase in a PrivacyProtected file with a strong password instead of on paper under your mattress.
PrivacyProtect government-issued IDs and health records before uploading them to cloud storage to keep Big Tech at bay.
It's possible to lose access to the device that generates your two-factor authentication (2FA) backup codes. It happened to me, and I'm locked out of my Dropbox account forever. Don't be like me. Store backup codes for things like your password manager and primary email account in multiple locations.
PrivacyProtect works on any device with a web browser, whether borrowed, brand new, or in a public library. Keep PrivacyProtected files on a USB drive and access them anywhere without installing software.
A non-obvious example: if your only goal is to keep Apple, Microsoft, etc., from reading your email or file, you could use a weak password like "123" and include the password itself in the hint. It still prevents their software from reading your file.
Related to the above, have you ever been pickpocketed or lost your passport in a foreign country? My friend lost his. He spent the day in the U.S. Embassy, stressed out, proving his identity. One of the items listed as "evidence of U.S. citizenship" on travel.state.gov is a photocopy of your missing passport. Skip (some of the) stress. Keep a PrivacyProtected file on your phone or host your document to access it in a pinch.

If you're short on time, you can skip the rest of this article and start using PrivacyProtect immediately; it's that simple! For those interested in details, read on.

How it works

Your secret is safe; it never leaves your device. No data transfers to or from PrivacyProtect servers after the initial page load. In fact, you can disable your internet connection, and concealing and revealing your secret still works. PrivacyProtect doesn't know who you are, what you're sharing, or who you're sharing it with.

For encryption, PrivacyProtect uses native browser APIs with no external dependencies. Argon2 doesn't have browser support. So to derive a key from the entered password, PrivacyProtect uses PBKDF2 with 2,100,000 iterations, a random salt, and the SHA-512 hash, as recommended here. This iteration count is ten times the OWASP-recommended 210,000. This is justified in light of the recent LastPass breach.

PrivacyProtect encrypts the plaintext using NIST-approved AES-256 in NIST-recommended GCM block cipher mode using the derived key and a random initialization vector. The HTML file contains the resulting ciphertext, initialization vector, and salt needed for decryption.

PrivacyProtect improves on Portable Secret, provides better protection than password-protected archives, and is free and open-source. Give it a try; I'd love to hear your feedback!

Connecting to Private RDS Databases From Your Local Machine

Rocky Warren — Tue, 17 Jan 2023 17:14:10 +0000

The following shows you how to connect to Relational Database Service (RDS) database instances in private Virtual Private Cloud (VPC) subnets from a local machine using EC2 bastion hosts and AWS Systems Manager (SSM) instead of using SSH directly. It assumes,

You have an EC2 instance running.
You have security groups configured so the EC2 instance has access to the RDS database.

If either of those isn't the case, follow those two steps in this article and then head back here.

If you let AWS store your database secrets in Secrets Manager when creating the database, all connection information is in the secret.

From the AWS Console

From the Secrets Manager Console, select to database secret.
Select Retrieve secret value, you'll need host, dbname, username, and password in the following steps.
From the EC2 Console, connect to the EC2 instance with access to your database using Session Manager.
From the launched terminal, run the following. This is specific to PostgreSQL, but you run equivalent commands for your database.

   # Install postgresql
   sudo amazon-linux-extras install -y postgresql14

   # Connect using the values copied in step 2
   # After entering it, you'll be prompted for the password
   psql -h [YOUR_HOST] -d [YOUR_DB_NAME] -U [YOUR_USERNAME] -W

   -- List tables
   \dt

   -- Run your favorite queries
   SELECT * FROM [YOUR_TABLE];

You're connected! But we can do better.

From the command line

Run the following to see if you have session-manager-plugin installed. If you don't, run the commands here to install it.

   session-manager-plugin --version

To start the SSM session, you need the EC2 instance ID. You can either get it from the AWS Console, or run the following with appropriate filters if you have lots of instances. The following assumes you have jq installed and have instance tags.

   aws ec2 describe-instances \
     --profile [YOUR_PROFILE] \
     --filter Name=tag:[YOUR_TAG_KEY],Values=[YOUR_TAG_VALUE] \
     | jq -r '.Reservations[].Instances[].InstanceId'

Given the EC2 instance ID, run the following to start the session,

   aws ssm start-session \
     --profile [YOUR_PROFILE] \
     --target [YOUR_INSTANCE_ID]

You're connected from your own terminal! Now, run the commands in step 4 above to connect to your database and start executing queries.

Getting MMS Working on GrapheneOS With Google Fi

Rocky Warren — Fri, 13 Jan 2023 00:00:00 +0000

I recently installed GrapheneOS on my Pixel and enjoy the privacy protection. I use Google Fi and could send MMS messages just fine. But when someone else in the group would send a message, I would see the error "MMS delivery failure; please contact Google Fi Support". Google Fi tech support was unable to help since I'm using a non-standard version of Android. Here's the solution.

First, I left the owner profile untouched and installed Sandboxed Google Play into a new profile; this will not work. When you attempt to activate Google Fi, you'll see the error "Account access only." From this support page, a troubleshooting step mentions that only the device owner can activate Google Fi service.

Installing Sandboxed Google Play onto the owner profile got me further, but I'd see "Your SIM card couldn't be read." The final piece to the puzzle is to go to Settings > Network & internet and toggle on the Enable privileged eSIM management setting. Now that I know what to search for, I found this discussion thread and this documentation explaining this in more detail.

Once toggled, open the Google Fi app to complete activation. If you're still having issues, this these steps,

Open Google Fi's App Info > Storage & cache
Press Clear cache and Clear storage
Back up a screen and press Force stop
Open Settings > System > Reset options
Press Erase downloaded SIMs
Reboot the device
Open the Google Fi app and activate

Once activated, you can toggle Enable privileged eSIM management back off. I hope this helps. Enjoy MMS and group texting!

A (Relatively Easy To Understand) Primer on Multi-Party Computation

Rocky Warren — Thu, 27 Oct 2022 00:00:00 +0000

Background: digital signatures

Blockchains use digital signatures, which use key pairs: a private key (PK) and a public key. Through digital signatures, any person with a PK can sign transactions to spend digital currencies and, assuming the PK has access, control smart contracts. Therefore, it is crucial to safeguard PKs. Some users safeguard PKs themselves and accept the risk of theft or loss. Others trust online wallets or exchanges to protect their keys.

In both these options, the user puts all their trust in a single entity. In contrast, threshold signature schemes (TSS) require a threshold of at least two cooperating participants to produce a signature.

Both multi-party computation (MPC) and multi-signature, or multi-sig, offer these capabilities. The resulting MPC signature is interchangeable with a single-signer signature. However, with multi-sig, each party has a PK. This slight difference significantly impacts the cost, speed, and availability on blockchains. For availability specifically, each blockchain requires multi-sig support either natively or via smart contracts. Supporting these increases development efforts for each new blockchain.

MPC overview

MPC is a sub-field of cryptography allowing parties to jointly compute a function over their inputs while keeping those inputs private. Unlike traditional cryptographic tasks that assure the security and integrity of communication and where the adversary is an outside party, this model protects party member privacy from each other.

The fundamental goals of an MPC protocol are,

Input privacy: one cannot infer any information about the private data held by the parties from the messages sent during protocol execution.
Correctness: any subset of colluding parties willing to share information or deviate from protocol execution instructions should not be able to force honest parties to output an incorrect result. This goal comes in two flavors: either the legitimate parties are guaranteed to compute the correct output (a "robust" protocol), or they abort if they find an error (a protocol "with abort").

A threshold signing algorithm has three phases,

Generate the key pair, split the PK into multiple secret shares, and distribute these shares between parties. Distributed key generation ensures each node only learns its share; the full key is never assembled or available in one place.
Gather a threshold of $t$ parties and run an MPC protocol to sign the transaction.
Verify the signature using standard signature verification algorithms.

A naive example

Three coworkers, Alice, Bob, and Carol, want to know their average salary without disclosing their salaries. The function they wish to compute jointly, then, is, $AVG(S_A, S_B, S_C)$ .

If they had a trusted friend, Dave, who they knew could keep a secret, they could each tell him their salary. Dave could compute the average and tell all of them. The goal of MPC is to design a protocol where, by exchanging messages only with each other, Alice, Bob, and Carol can still learn the average without revealing who makes what and without relying on Dave.

Alice picks a random number known only to her and adds it to her salary. She then securely passes the result to Bob. Bob adds his salary to the first result and gives the result to Carol. Carol does the same and returns the result to Alice. Alice then subtracts the random number from the total and calculates the average salary. The parties can only learn their input and the output, the exact information they would learn if trusting their honest friend, Dave.

That's MPC in a nutshell. In practice, it's far more complex for various reasons (such as robustness in the face of collusion and malicious actors).

Example use cases

Distributed voting
Private bidding and auctions
Sharing of signature or decryption functions
Private information retrieval (allowing a user to retrieve an item from a server in possession of a database without revealing which item was retrieved)

Key pair generation with secret sharing

A system is not decentralized if one entity controls a client's PK. MPC nodes, or nodes for short, own only one piece of a PK, known as a key share. The nodes then work together using an MPC protocol to perform necessary computations.

Secret sharing protects against node failures. Secret sharing is a method for distributing a secret among a group so that no individual holds any intelligible information about the secret. However, a sufficient number of individuals may combine their shares to reconstruct the secret. Any group of $t$ (for threshold) or more can together reconstruct the secret, but no group of fewer than $t$ can. Such a system is called a $(t, n)$ -threshold scheme.

Consider the secret sharing scheme where $X$ is the secret to be shared, $P_i$ are public asymmetric encryption keys, and $Q_i$ are their corresponding PKs. Each party $J$ is provided with ${P_1 (P_2 (...(P_N (X )))), Q_j\}$ . In this scheme, any party with PK one can remove the outer layer of encryption, a party with PKs one and two can remove the first and second layers, and so on. A party with fewer than $N$ keys can never fully reach the secret $X$ without first needing to decrypt a public-key-encrypted blob for which they do not have the corresponding PK. This problem is currently computationally infeasible.

Shamir's Secret Sharing is a popular scheme. In contrast to the scheme above, one may recover the secret with any $t$ out of $n$ shares instead of requiring all $n$ . Based on the Lagrange interpolation theorem, the essential idea is that $t$ points are enough to uniquely determine a polynomial of degree less than or equal to $t - 1$ . For instance, two points are sufficient to define a line, three are enough to represent a parabola, and so on. The method is to create a polynomial of degree $t - 1$ with the secret as the first coefficient; pick the remaining coefficients randomly. Then, find $n$ points on the curve and give one to each party. When at least $t$ out of the $n$ parties reveal their points, there is sufficient information to fit a $(t - 1)$ th degree polynomial, with the first coefficient being the secret. If this is unclear, see this blog post for an alternate explanation.

It’s possible to draw an infinite number of polynomials of degree two through two points. Three points are required to determine it uniquely.

Each MPC node receives only one PK share and uses it to construct partial transaction signatures to broadcast to the network. Even if a node is offline, loses PK access, or is malicious, it is still possible to reconstruct the complete transaction from the remaining nodes.

The Importance of Cultivating a Deep Work Culture

Rocky Warren — Fri, 07 Oct 2022 00:00:00 +0000

Protected "maker", "flow", or "deep work" time is important to productivity. It's difficult for larger companies to change their culture around meetings, it's not for startups. Implementing a deep work culture and fostering it as companies grow gives them a strong competitive advantage: increased per-employee productivity. This means delivering high value to customers with relatively low headcount. A low headcount, in turn, keeps communication overhead low, creating a positive flywheel.

Research

Snippets from Maker's Schedule, Manager's Schedule by Paul Graham,

There are two types of schedule, which I'll call the manager's schedule and the maker's schedule. The manager's schedule is for bosses. It's embodied in the traditional appointment book, with each day cut into one hour intervals. You can block off several hours for a single task if you need to, but by default you change what you're doing every hour.

When you use time that way, it's merely a practical problem to meet with someone. Find an open slot in your schedule, book them, and you're done.

Most powerful people are on the manager's schedule. It's the schedule of command. But there's another way of using time that's common among people who make things, like programmers and writers. They generally prefer to use time in units of half a day at least. You can't write or program well in units of an hour. That's barely enough time to get started.

When you're operating on the maker's schedule, meetings are a disaster. A single meeting can blow a whole afternoon, by breaking it into two pieces each too small to do anything hard in. Plus you have to remember to go to the meeting. That's no problem for someone on the manager's schedule. There's always something coming on the next hour; the only question is what. But when someone on the maker's schedule has a meeting, they have to think about it.

I find one meeting can sometimes affect a whole day. A meeting commonly blows at least half a day, by breaking up a morning or afternoon. But in addition there's sometimes a cascading effect. If I know the afternoon is going to be broken up, I'm slightly less likely to start something ambitious in the morning. I know this may sound oversensitive, but if you're a maker, think of your own case. Don't your spirits rise at the thought of having an entire day free to work, with no appointments at all? Well, that means your spirits are correspondingly depressed when you don't. And ambitious projects are by definition close to the limits of your capacity. A small decrease in morale is enough to kill them off.

Each type of schedule works fine by itself. Problems arise when they meet. Since most powerful people operate on the manager's schedule, they're in a position to make everyone resonate at their frequency if they want to. But the smarter ones restrain themselves, if they know that some of the people working for them need long chunks of time to work in.

Those of us on the maker's schedule are willing to compromise. We know we have to have some number of meetings. All we ask from those on the manager's schedule is that they understand the cost.

From a summary of Deep Work by Cal Newport,

Deep work is the ability to focus without distraction on a cognitively demanding task. It's a skill that allows you to quickly master complicated information and produce better results in less time. Deep work will make you better at what you do and provide the sense of true fulfillment that comes from craftsmanship. In short, deep work is like a super power in our increasingly competitive twenty-first century economy. And yet, most people have lost the ability to go deep—spending their days instead in a frantic blur of e-mail and social media, not even realizing there's a better way.

From a summary of A World Without Email by Cal Newport (an unfortunate title, but he specifically and repeatedly mentions Slack in the book as well),

Modern knowledge workers communicate constantly. Their days are defined by a relentless barrage of incoming messages and back-and-forth digital conversations–a state of constant, anxious chatter in which nobody can disconnect, and so nobody has the cognitive bandwidth to perform substantive work. There was a time when tools like email felt cutting edge, but a thorough review of current evidence reveals that the "hyperactive hive mind" workflow they helped create has become a productivity disaster, reducing profitability and perhaps even slowing overall economic growth. Equally worrisome, it makes us miserable. Humans are simply not wired for constant digital communication.

We have become so used to an inbox-driven workday that it's hard to imagine alternatives. But they do exist. Drawing on years of investigative reporting, author and computer science professor Cal Newport makes the case that our current approach to work is broken, then lays out a series of principles and concrete instructions for fixing it. In A World without Email, he argues for a workplace in which clear processes–not haphazard messaging–define how tasks are identified, assigned and reviewed. Each person works on fewer things (but does them better), and aggressive investment in support reduces the ever-increasing burden of administrative tasks. Above all else, important communication is streamlined, and inboxes and chat channels are no longer central to how work unfolds.

Finally, from a summary of It Doesn't Have to be Crazy at Work by Jason Fried and David Heinemeier Hansson,

Calm is meetings as a last resort. Calm is contextual communication. Calm is asynchronous first, real-time second. Calm is more independence, less interdependence.

Takeaways

Startups can cultivate deep work in the following ways,

Favor asynchronous communication in ticket trackers (e.g., Trello, Linear, JIRA) over synchronous Slack.
- Ticket trackers should contain all context for specific features and projects (including links to any documents and meeting notes on the subject) rather than having to search for piecemeal conversations in Slack.
- In this way, picking up a new story gives you everything you need to know to accomplish it. This may require breaking stories down in more detail so there are fewer implementation questions.
Use Clockwise or similar to aid in meeting scheduling.
- It takes each person's calendar into consideration to group meetings into chunks as much as possible, leaving uninterrupted blocks of time for deep work.
- There is an option to block off "Focus Time" on your calendar and it integrates with Slack to turn on "Do not disturb" during these times so you don't get distracted by Slack notification noise.
- Understand this may mean delayed Slack responses. That should be okay! Allow people to remain in deep work and respond when they've come up for air.
At least one, ideally two "No meeting days" for makers and optionally managers.
- No cadence meetings scheduled on these days
- Slack standup instead of in-person or over video chat
- Impromptu meetings for blockers are okay, but try to have multiple tickets to work on so you can move onto another if you get stuck.
- As another option in place of "No meeting days", have bi-modal days where the mornings are for meetings and the afternoons are for deep work (or vice versa).
Meeting invites must include GAP: goals, an agenda, and pre-meeting preparation. If they don't, you can skip the meeting — no GAP, no need to attend.

If the company buys into this way of working, the ROI is potentially large. It's a mindset shift. But for me personally, my best work is done when I lose all track of time, forget to eat, and come out on the other side with shit fucking done. Imagine if an entire company could optimize their days to get as much of that time as possible. Your competition is in trouble.

Death to (Synchronous) Standups

Rocky Warren — Tue, 04 May 2021 00:00:00 +0000

Some back of the napkin math no one asked for about how much daily standups cost your team compared to asynchronous standup tools like Geekbot.

Assuming 20 mins per standup (this could be an understatement) and five mins for Geekbot (usually an overstatement), we'll say async standups save 15 mins/day per person.
Assuming a standup including ten people, that's 150 mins/day or 2.5 hrs/day, 12.5 hrs/wk, and ~625 hrs/yr (assuming 50 weeks worked).
625 hrs/yr is over 15.625 weeks(!) of time per year spent in standup or nearly eight full sprints. So just under a sprint per person per year spent in this single meeting.
Assuming a conservative average of $50/hr, that's over $30,000/yr. But this number pails in comparison to the opportunity cost of an extra sprint each year you could instead build products for customers.

Especially with more workers going remote, a common argument is that the meeting is more about human interaction. But I'd argue a dedicated, weekly meeting to discuss team member's weekends, etc. along with regularly scheduled happy hours work better. These can lead to deeper conversations and more positivity than hearing each other's voices each morning give status updates.

In addition to the time savings, Slack-style standups give these added benefits:

A historical record of what everyone said they were going to do and what they actually did.
The productivity benefits of taking time to plan your day and get it in writing, helping to 1) stay accountable to yourself and others and 2) reduce distracting rabbit holes.
Avoid the context switch (which studies have shown adds 15 mins and double my numbers above) of a meeting each morning and instead do your standup report when you're not in the middle of focused time.

It's up to each team to decide if an async standup is right for them. Hopefully, these numbers show the expense of daily standups. Make sure the benefits are outweighing these costs.

OpenAPI Spec-First API Development

Rocky Warren — Wed, 07 Apr 2021 00:00:00 +0000

Recent APIs I've built follow both the OpenAPI (formerly known as Swagger) and JSON:API specs. The former allows us to generate server-side data models and API clients in a number of languages using the OpenAPI Generator. The latter allows us to leverage shared conventions on how a RESTful JSON API should function.

Through much trial and error, we've landed on a solid approach to automating much of our workflow when adding new API endpoints. The process begins as follows,

Update the OpenAPI schema
Use Vert.x Web API Contract to import the schema to handle things like authentication and request validation
Use the OpenAPI generator to create server-side domain models and implement the endpoint

Since we offer Postman collections for our API, we chose to host our documentation using Postman as well. Ideally, we'd upload the new schema during our CI process, which would trigger the collection and documentation to update. Unfortunately, Postman wasn't built with this workflow in mind.

They allow you to upload an OpenAPI schema and generate a collection from it, but not through their API. This means we can't automate it. So as a workaround, we use the openapi-to-postman tool to convert our schema to a Postman collection and then use their API to update the collection we provide our customers.

A big problem with this tool, however, is that it generates new IDs for endpoints each time you run it. This makes diffs hard to follow and, more importantly, breaks documentation URLs. To workaround that, we use a forked version that stores a sourcemap of the endpoints so the IDs remain stable.

From there, we call Update Schema with the updated OpenAPI schema. This doesn't update Postman collections created from the schema, however, so we then call Update Collection, which, thankfully, triggers the documentation to update.

Now that our Postman collection, documentation, and code samples are updated, we use the OpenAPI generator to update our API clients and we're done.

This seems like a relatively normal use-case: update the OpenAPI spec and have all downstream artifacts ("Golden" Collection, Documentation, Forked Collections, API clients) automatically update to reflect those changes. If you'd like to see Postman handle this better, add your thoughts to this pull request.

Rust: First Impressions

Rocky Warren — Thu, 04 Mar 2021 00:00:00 +0000

Scala is great for recent microservices I've written, enabling incredible velocity. However, certain scenarios benefit from lower-level languages.

I have five years of (dated) C++ experience. It's a high learning curve and even experts can cause segfaults, undefined behavior, double free errors, array overruns, mishandle concurrency, etc. This prevents novices from jumping into code for fear of breaking things, slowing productivity. Even modern C++ lacks a strong tooling ecosystem, making some things, including AWS Lambdas, a "batteries not included" experience.

For these reasons, I spent time looking at how Rust and Go compare. The key requirements,

Integrates with existing code, libraries, and tools
Increases productivity
Reduces bugs by catching preventable ones early and being hard to misuse
Negligible performance differences
Comparable learning curve (ideally, shorter)
Doesn't impact recruiting (ideally, improves)
Stable with backward compatibility, meets future needs

Both Rust and Go are great, but have different priorities as this article describes. Compared to Go, Rust has better C++ interoperability, no garbage collector (leading to better and more consistent performance), and richer language features (at the cost of additional complexity/higher learning curve). Taken together, these fit my use cases and preferences better.

Addressing the "Key Requirements" above, Rust has

Great C++ interoperability via the CXX crate
an aim to be faster than C
Excellent tooling
- Cargo provides compile, test, format, lint, package management, and workspace/monorepo support
- VS Code extensions provide the above along with in-editor line coverage
A sophisticated compiler giving non-experts the confidence to make changes since it won't - build code with mistakes easy to make in C++
Concurrency via familiar async/await
Zero-cost abstractions around things like generics
Functional programming features like iterators and closures
A devoted following as Stack Overflow's Most Loved Language five years running
A bright future. As our front-end apps grow in complexity, Rust's relative approachability and WASM compilation may become important. Other languages have this capability, but at least at this point garbage collection adds client-side baggage.

Additional resources include,

Running Linux GUI Apps in Docker on Mac

Rocky Warren — Tue, 02 Feb 2021 00:00:00 +0000

Why would you want to run Linux apps in Docker on a Mac? Maybe there's a Linux program you love or you want the added security of sandboxing an app. Or maybe, as was the case with me, you're developing a C++ program targeting Linux and don't want to cross compile your dependencies.

The following steps get the CLion IDE running in a Docker container created during the project's build process. It uses XQuartz, a version of the X Window System that runs on macOS, along with socat.

Download the Linux version of CLion
Each time dependencies change, run create_dev_container.sh to create a Docker image and container. The script assumes ~/dev is your code directory. If you use something different, edit the script or run it as CODE_DIR=<your_dir> ./create_dev_container.sh
Run start_dev_container.sh to start the container and exec bash inside of it
From inside the container, run /home/dev/clion-XXX/bin/clion.sh to start CLion. If this fails, ensure your container contains a JRE by either manually running apt install default-jre or adding it to the Dockerfile.
Either select "Evaluate for free" or enter your activation code
Open your project
Build it and enjoy code completion, etc.!
If you make changes to settings they'll be lost each time you create a new container. To save them, run docker cp my_project:/root/.CLionXXX/ ~/dev and import them next time you start CLion in a fresh container.

create\_dev\_container.sh:

NAME=my_project
EXISTING_IMAGE_ID=$(docker ps --all --filter "name=$NAME" --format "{{.Image}}")

if [[-n "$EXISTING_IMAGE_ID"]]; then
  read -p "$NAME already exists. Delete it? " answer

  while true
  do
    case $answer in
    [yY]* ) docker stop $NAME > /dev/null 2>&1
            docker rm $NAME > /dev/null 2>&1
            docker rmi $EXISTING_IMAGE_ID
            break;;
    * ) echo "exiting"
            exit;;
    esac
  done
else
  echo "$NAME does not exist, creating..."
fi

# Build your docker container
./build.sh

CODE_DIR=${CODE_DIR:-~/dev/}
IP_ADDRESS=$(ifconfig en0 | grep "inet " | cut -d " " -f 2)
IMAGE_ID=$(docker images --format "{{.ID}}" build/local/my-project:latest)

echo "Creating container using image ID $IMAGE_ID, mapping IP address $IP_ADDRESS, and mounting $CODE_DIR..."
docker create -it -e DISPLAY=$IP_ADDRESS:0 -v $CODE_DIR:/home/dev/ $IMAGE_ID bash

GENERATED_NAME=$(docker ps --all --filter "ancestor=$IMAGE_ID" --format "{{.Names}}")
echo "Renaming $GENERATED_NAME to $NAME..."
docker rename $GENERATED_NAME $NAME

echo "\nContainer created, run ./start_dev_container.sh to exec bash inside of it."

start\_dev\_container.sh:

NAME=my_project

APP=socat
brew ls --versions $APP > /dev/null 2>&1 || brew install $APP
APP=xquartz
brew cask ls --versions $APP > /dev/null 2>&1 || brew cask install $APP

if ["$(lsof -nP -i4TCP:6000 | grep -c LISTEN)" -eq 0]; then
  socat TCP-LISTEN:6000,reuseaddr,fork UNIX-CLIENT:\"$DISPLAY\"
fi

open -a Xquartz
docker start $NAME > /dev/null 2>&1
docker exec -it $NAME bash

Start Local Services Fast in Zero Lines of Code with Docker Compose

Rocky Warren — Mon, 04 Jan 2021 00:00:00 +0000

...unless YAML counts as code.

Many companies have home-grown scripts and tools to start services locally. In my experience, the scripts start as well-intentioned wrappers simplifying common docker and docker-compose commands. They tend to age poorly, however. They get complex, are a maintenance burden, and force developers to learn the wrapper commands instead of the underlying tool commands they may already be familiar with. They don't wrap each flag or option, making them less flexible. When the tools they wrap update, the tool must change to take advantage of new features. This, in turn, can lead to using deprecated, slow, or less secure commands, impacting the engineering team's ability to deliver value to customers.

Twice I have converted internal tools to use docker-compose directly. Using this approach enables,

YAML's merge syntax for sharing common configuration between services
Showing all services and their dependencies in once place (or multiple, depending on your use-case)
Quickly iterating on configuration changes
Relying on robust, community-supported CLI with no maintenance requirements
Using new features on day one instead having to add support
Exporting the COMPOSE_FILE variable so you can start services from any directory.

We'll walk through an example docker-compose.yml file using this approach.

version: "2.4"

x-depends-on-postgres: &depends-on-postgres
  postgres: { condition: service_healthy }

x-interval5-retries45-timeout5
  interval: 5s
  retries: 45
  timeout: 5s

First, note we use v2.4 syntax instead of v3.x. Think of docker-compose v2 and v3 more like a fork than an upgrade, with v2 perfectly capable provided you don't need Docker Swarm support. Ideally, your services are resilient to dependencies being down, allowing you to start every service in parallel. If they aren't, docker-compose v2.4 syntax allows using depends_on conditions and healthcheck keys together, allowing you to start services in parallel if they're not dependent on each other and in order if they are. If you're using v3.x, this page gives options to accomplish something similar, they're just more work and, sadly, there's no plan to add the v2 functionality back.

Next, we add common configuration using keys that start with x- via YAML merge key types, which docker-compose supports. This example only has two services so not much is shared, but this can lead to a large reduction in duplication for more realistic scenarios.

services:
  my-service:
    depends_on:
      <<: *depends-on-postgres
    environment:
      SERVICE_NAME: my-service
    image: ${MY_SERVICE_REPO:-MY_DEFAULT_REGISTRY}/my-service:latest

  postgres:
    environment:
      PGDATA: /var/lib/postgresql/data/pgdata
      POSTGRES_DATABASE: my-service-data
      POSTGRES_PASSWORD: xxx
      POSTGRES_USER: user
    healthcheck:
      test:
        "res=$$(echo 'SELECT 1' | psql --host localhost --username $$POSTGRES_USER
        --dbname $$POSTGRES_DATABASE --quiet --no-align --tuples-only) &&
        [$$res = '1'] && exit 0 || exit 1"
      <<: *interval5-retries45-timeout5
    image: postgres:12
    ports: [5432:5432]
    volumes:
      - ./data/postgres:/var/lib/postgresql/data/pgdata:delegated
    command: postgres

In the services section, add each as you normally would. Then, specify the depends_on key for any service with dependencies and the healthcheck key for any service depended on by another service. To add a healthcheck,

Many services may share a healthcheck, in that case, pull it into a YAML merge key type to reduce duplication
For new or one-off services, start by looking here to see if one exists or for inspiration. Once you have a command you think works, exec into the container with docker-compose exec SERVICE sh and test it. Some containers won't have curl or other tools, so you may need to get creative. Once it's working, add the command to docker-compose.yml, start the service, and run docker-compose ps to ensure it becomes healthy. If it doesn't, run docker inspect $(docker-compose ps -q SERVICE) | jq '.[].State.Health' to see what's being returned, edit the command, and try again.

You'll notice the image for my-service's image is ${MY_SERVICE_REPO:-MY_DEFAULT_REGISTRY}/my-service:latest. This allows quickly swapping between local and hosted images using a .env file. By default, it uses MY_DEFAULT_REGISTRY. To use a local image, add MY_SERVICE_REPO=local to .env.

services:
  # ...as shown above

  flow-backend:
    depends_on:
      my-service: { condition: service_healthy }
      # ...other services
    image: rwgrim/docker-noop

You'll likely have services you tend to start together. To make this easier, optionally add "flows" at the bottom of services. These only exist to start their dependents, so they use the tiny rwgrim/docker-noop image, which simply exits successfully. In a real application, maybe you'll have flow-backend, flow-frontend, and then a flow-all that depends on both of the previous flows.

Here's the full example,

version: "2.4" # v3 doesn't support using depends_on conditions with healthcheck

#############################################################################
# Shared config via merge key types, https://yaml.org/type/merge.html
#############################################################################
x-depends-on-postgres: &depends-on-postgres
  postgres: { condition: service_healthy }

x-interval5-retries45-timeout5
  interval: 5s
  retries: 45
  timeout: 5s

services:
  #############################################################################
  # Services
  #############################################################################
  my-service:
    depends_on:
      <<: *depends-on-postgres
    environment:
      SERVICE_NAME: my-service
    image: ${MY_SERVICE_REPO:-MY_DEFAULT_REGISTRY}/my-service:latest

  postgres:
    environment:
      PGDATA: /var/lib/postgresql/data/pgdata
      POSTGRES_DATABASE: my-service-data
      POSTGRES_PASSWORD: xxx
      POSTGRES_USER: user
    healthcheck:
      test:
        "res=$$(echo 'SELECT 1' | psql --host localhost --username $$POSTGRES_USER
        --dbname $$POSTGRES_DATABASE --quiet --no-align --tuples-only) &&
        [$$res = '1'] && exit 0 || exit 1"
      <<: *interval5-retries45-timeout5
    image: postgres:12
    ports: [5432:5432]
    volumes:
      - ./data/postgres:/var/lib/postgresql/data/pgdata:delegated
    command: postgres

  #############################################################################
  # Flows
  #############################################################################
  flow-backend:
    depends_on:
      my-service: { condition: service_healthy }
      # ...other services
    image: rwgrim/docker-noop

For even higher productivity, export COMPOSE_FILE='path/to/docker-compose.yml' and create the alias alias dcp=docker-compose, allowing you to start services from any directory!

Day-to-day, you'll use docker-compose commands directly. Common commands: run dcp up -d SERVICE_1 SERVICE_2 ... to create and start the provided services and their dependencies (the -d or --detach flag runs them in the background), dcp ps to list their statuses, dcp stop to stop them, dcp start to start them back up, and dcp down to remove containers, images, and volumes. Run dcp pull to download the latest images, dcp logs SERVICE_1 SERVICE_2 to view logs, dcp config --services to validate the docker-compose.yml file and list all services defined, and dcp config --services | grep flow to list only flows. Run dcp -h for all commands.

And that's it! All services configured in one file with minimal duplication that you can run anywhere using the newest features of a community-supported CLI with no code to maintain!

Git Fundamentals

Rocky Warren — Tue, 01 Dec 2020 00:00:00 +0000

Git can seem overwhelming, but while performing typical day-to-day tasks, you'll use only about ten commands. If you happen to use Oh My ZSH, the included git plugin adds many aliases. Below each command, I've included the corresponding alias.

# Start a new branch for a new feature
git checkout -b new-feature
gcb new-feature

# Commit as you make changes (the --all adds changed files to
# `staging`)
git commit --all --message "your commit message"
gcam "your commit message"

# Add new or deleted files to `staging` (the `git commit
# --all` above only adds tracked files)
git add --all
gaa

# Check the status of your branch/commits
git status
gst

# Switch branches
git checkout branch-name
gco branch-name
# Separate `git` plugin alias to switch back to master
gcm

# See what has changed (maybe done prior to committing, include
# the `--cached` flag to see what has changed in `staging`)
git diff
gd

# Pull new changes down from remote (Github/GitLab/etc.)
git pull
gl

# Push changes to remote
git push --set-upstream origin
gpsup

# Merge changes from another branch into the current one
git merge other-branch-name
gm other-branch-name

# Or, if you prefer rebase,
git rebase -i other-branch-name
grbi other-branch-name

And that's it! 95% of the time, you'll use these commands. Google was invented for the remaining 5% 😉