DEV Community: Jean-Mark Wright

The story behind Wave's boring Monetization release

Jean-Mark Wright — Tue, 19 Mar 2024 13:18:46 +0000

Original artlce: https://jaywhy13.hashnode.dev/the-story-behind-waves-boring-monetization-release

Introduction

On January 29th this year, a seemingly grandiose thing happened. We started monetizing some core features in our 10+ year old product. Beginning then we invited new customers to purchase a subscription to enjoy some exclusive features in our product that existing users enjoyed for free. The launch itself was overwhelmingly uneventful, and admittedly, somewhat anticlimatical. It was however, successful. This success was due in large part to the fact that, we didn't do anything big on the day of the launch. The heavy lifting was done months before... And that's what this story captures.

So we wanted to add a price tag to some features in our product... How hard could that be? Couldn't we just get a marker and someone with great penmanship and get it over with? Hmmm... lemme take that one to the project retrospective. In the mean time! Take a product and a company that's both over 10 years old and you've got an entire ecosystem that likely isn't ready for your new grand vision. The passage of time shapes and skews your system with a particular slant. The system is a wonderful mix of customer features, best practices, mix in some anti-patterns and technical debt. All of it contributes in some way to friction that makes fundamental, cross-cutting changes much more interesting... where "interesting" could be anywhere from exciting, thrilling, to overwhelming, harrowing or near impossible.

In our case, our age forced us to carefully work through several technical and non-technical challenges to make our dream a reality. Many departments outside Engineering became key stakeholders. Our customers would need extra support from our Customer Experience team as we transitioned. The team also needed to understand and identify paying customers in the systems they used. One of our adjunct departments that provided coaching services for customers would be impacted by our regulation of premium features. Conversations and solutions would be necessary to prevent disruption for coaching clients. The list of departments, dependencies and interconnections grew as the project aged.

Inside Engineering, almost all our teams had a role to play in the project. Our product teams needed to make changes to incorporate and enforce the distinction between paid and free features. Our frontend and mobile teams would work on the the experience that allowed customers to purchase a subscription and gain access to premium features. Our API team needed to enable the regulation of access to premium features. And well, we needed a net-new system to regulate access to these premium features -- that's where our team came in! Also, also... our Data team would need information from our system to provide reports for our Marketing team and other areas of the business.

For our grandiose monetization project, we introduced a net-new service that regulated access to premium features. The introduction of this service created a new dependency that sat at the centre of all our products and many departments too. We worked with multiple departments, micro-services, technical debt, scope reductions to realize our monetization vision. It was no easy feat, but it's a beautiful story to reflect on.

Everything before the launch

Well... that was one long introduction! Let's transition to talking about what we did before the launch, and the steps that were critical to the success of the launch. First, I'll describe the new microservice we built and the use cases it was intended to service. Next, I'll walk through the timeline of the project, highlighting the strategies we took that helped us de-risk the launch.

The newest service around town -- Are you entitled?

Our team was tasked with building a new microservice that was responsible for regulating access for premium features and facilitating the purchase of subscriptions that provided access to said features. We called the service the "Entitlements" service because of the dominant use case of determining whether a business is "entitled" to a certain feature. Our users could purchase a plan (which provides a set of features) for one or more of their businesses in Wave. Before purchasing a plan, users would be on a plan (STARTER plan) which gave access to some features. Purchasing a PRO subscription would give them access to more features.

Our system was responsible for answering questions like...

Does this business have access to this feature?
What plans can I purchase for my business?
What plan(s) do I currently have for my business?
Which subscriptions have I purchased for my business?

And our system would also facilitate, or perform the following tasks:

Put newly created businesses on a plan
Enable customers to purchase, upgrade or cancel a plan

So, with our new microservice we could get context on access businesses had to premium features and plans could be purchased.

The Entitlments service would also introduce some new interconnectedness into our existing microservices. Our product teams would rely on our service to provide sufficient context to present the appropriate experience for customers. Our invoicing product for example would need to show customers on our free (STARTER) plan nudges to access premium customer reminders. Our User Management feature which allowed our business owners to add other users to their business would need to represent some user types as premium ones. Our frontend and mobile teams would need to leverage our information on plans to show customers their current plan and upsell our premium plans. Our public API which proxies all traffic through to our backend microservices would need to provide capabilities for us to enforce the regulations of the Entitlements service. The Entitlements service represented a new core dependency for almost all our existing microservices.

What did we do?

Now let's traverse the timeline and call out some strategies that were crucial to the success of the project. It's mostly chronological, but there might be some overlap...

Consumer-driven design

Our first step was to meet with each of the teams (there were about eight of them) to understand their requirements so we could design, plan and prioritize. Each product team had a mandate crafted by their respective product leads for the changes they needed to make. This mandate would shape their interaction with the Entitlements service. Understandably, no product was identical but there were lots of commonalities. This information was critical for planning and determining where we place our focus.

In meeting with the teams we sought to understand interactions with our service, shared our initial API designs, then captured the dependencies they would use. In our initial meetings, we aimed to capture all interactions with our systems by framing questions around the subscription lifecycle along with other general questions. We asked questions like:

What actions does your service need to take when a business purchases, activates, cancels or resumes a subscription? This helped us understand what data teams needed to pull out of our system via an API and also data we needed to push to them (e.g. via event streaming using Kafka)
What information do you need to support your use cases in those specific scenarios? This helped us tease out the details of our APIs.
Where are your features implemented (e.g. in a backend service, in our single page app), and what libraries are being used (e.g. React, Redux). These questions helped us understood where abstractions were most needed. While our single page app is in React, there were important differences across the teams. Some teams were using different libraries for state management (e.g. Redux, local component state using GraphQL) and different component styles (e.g. older class-based vs functional components).
Are there legal, regulatory or compliance requirements that we need to consider? This helped us understand any constraints we'd need to bake into how our system worked. Our Payments team for example, ended up doing some extra checks to ensure there was no de-sync between our systems. They needed to ensure that the access our system described matched the access they were implementing on their side. This had monetary implications for customers.
Are there any risks or roadblocks you forsee that we'll need to mitigate? For example, are there any endpoints that are really high in traffic, or require low latency? Through these questions we uncovered a use case where our Accounting team needed to submit bulk, bursty traffic over small periods of time.

Once we had documented the responses for all of these questions and met with the teams, we could transition to designing capabilities. We separated the system into logical chunks and created tickets for engineers on the team to pick up. The team discussions provided a rich set of context that we transferred to the tickets, and could be used as acceptance criteria. This simplified the design process for our backend APIs and our frontend abstractions. We had a clear set of use cases we could reference. When these designs were completed, we met with the teams to show them our designs.

Meeting with the teams helped us determine the full set of capabilities we needed to build to satisfy all the use cases for the teams. From these conversations, we knew how many teams needed which capabilities, and why. Presenting these capabilities to the teams helped us validate the fulfilment of their requirements.

Fake it until you make it -- unlocking with contracts

After our APIs and frontend abstractions were designed, we opted to build fake capabilities to unlock the teams and save time. Recall, there were about eight teams that were dependent on our capabilities. Eight different teams meant lots of teams to work with and confirm that things actually worked when we implemented them. Additionally, while most of the teams would use a core set of capabilities, some teams needed some more nuanced capabilities to satisfy their requirements. It just wasn't practical for us to let all the teams wait until we had built both the common capabilities and more nuanced ones. It would take too long!

This is where things got interesting! We were able to go to production with these fakes months before the actual launch. We leveraged one of our product requirements that made this possible -- our existing users would continue to have the same experience after our launch. Our Product team decided that we would consider all our pre-launch users as "legacy" users, and they would retain access to the monetized features for free. When we launched, we would draw a line in the sand and all new customers would have to pay for the features we now deemed premium. However, nothing would change for pre-launch users. They would continue to enjoy Wave as they did before the launch.

Ok... big deal! So we decided to allow our existing customers to avoid paying for the premium features. What did that mean for our capabilities. That meant we had until the launch to fake the "legacy" experience. All our capabilities could statically treat all users as though they were legacy users. So, we could fake our entire backend APIs on the frontend by faking the legacy experience. In more concrete terms:

Our API that answered the question "is the business entitled to X feature" would always return True.
The API that provided the list of features the business has access to would always return the full list of features.
Our API that returned informations on plans one could purchase would always return a static set that are available to our legacy users.

So faked it we did! All of our frontend abstractions and backend APIs initially returned a canned list of responses. The frontend abstractions didn't need to hit our backend APIs and our backend APIs never needed a database or any other persistence mechanism. Fake, fake, fake! We built React hooks and higher order components that returned fake responses. Some teams needed to hit our backend APIs. For those, we built the services and repository layers, but they all fetched data from constants. We also provided some knobs that allowed teams to configure our fakes. This allowed teams to test different scenarios manually, or using automated tests.

Building the fakes provided a trove of benefits! For starters, it allowed all the other teams to start working in parallel while our team started building out the real implementations. It meant we could start to get feedback early about bugs, and interaction issues with our service -- lots of early validation. It also saved time! The teams were mostly able to finish off their implementation before we were through with ours. It also provided really high confidence that the use cases worked! Teams were able to go to production once they had completed their use cases! Faking our capabilities using contracts really helped optimize the timeline and provided great feedback for the entire implementation.

Enough faking -- the launches before the launch

The real launches took place as we gradually removed the fakes we had in production. Internally we crafted a detailed rollout plan, that specified how and when we would remove the fake capabilities on the frontend and the backend. It was something of a masterpiece! We started with providing access for premium features, then enabled the purchase flow, which ultimately prepared us for the external launches to Canada and the US. Additionally, we did some technical design that reduced the likelihood of regressions and mistakes during surgical removal of fakes.

Our rollout plan outlined that we would prioritize access information for features, then move on to the purchase flow. This meant all our products could implement the distinctions in functionality for premium flows. Our Invoicing products, for instance, could show sending automated reminders as premium for businesses without access to the premium feature. These calls would now go straight to our backend, through our frontend abstractions. Additionally, teams that hit our backends directly would benefit from responses that were backed by persistence and real logic. Focusing on the purchase flow enabled us to simulate paying customers. We had APIs that provide information on what plans a customer could purchase, and others actions like cancellation.

We also took extra caution to ensure we didn't mess up the fake removal surgeries. On the backend, we implemented a proxy repository pattern that enabled a clean separation of the fake from the real implementation. All calls to our persistence were proxied through this component which directed calls to our fake or real implementation. We took a similar approach on the frontend with some differences. The goal was to ensure that it was crystal clear, what was being faked and what was not. Largely, this extra work paid off as we were able to avoid regressions and didn't have major bugs or incidents while we swapped out the fake implementations.

All in all, we did several mini-launches which greatly de-risked the entire initiative. Each fake we removed meant we were closer to the application behaviour we needed for our external launch. All our capabilities were built and functioning in production, and our product teams were actively using those capabilities. Having our capabilities being actively used provided a high confidence that they worked as they should. It also allowed us to resolve some minor issues that came up. These launches represented the proverbial tree that fell in the forest.... No one knew!

So the actual launch then?

Fast forward to our launch date, on Jan 29 -- what did we actually do? Well... all we did was to change a configuration for how we treated users that signed up and the businesses they created. Prior to our launch date, all businesses created would be considered legacy. This meant users could enjoy our premium features for free. We would assign the necessary access (entitlements) to enable the business to access all our features. Being on a legacy plan disqualified users from purchasing our premium plans since they had all the features of the premium plan.

When we launched, we stopped doing that. After the launch, we flipped a switch and users and their businesses would now get free access to a smaller set of features. Additionally, they would now be eligible to purchase our premium plans. This eligibility enabled nudges sprinkled in each product to upsell our premium plans. And ahh... that's it! That's all we did! After we launch, we effectively stopped giving the legacy experience away, but instead provided less access. The heavy lifting was done months before when we removed our fakes and started using the system in production!

Conclusion

The journey of monetizing some core features of our decade-old product is a beautiful success story of continuous delivery. The essence of our success wasn't attributed to the steps on the launch day but rather the methodical, behind-the-scenes orchestration that allowed us to seamlessly transition to a monetized model without disrupting the user experience. The real work went in long before the curtains were raised. As we move forward, the lessons learned from this initiative will undoubtedly shape our approach to future projects. The successful launch of our monetization strategy serves not just as a milestone but as a beacon, guiding us towards continued innovation, customer engagement, and operational excellence.

I hope this narrative inspires you as embark on similar journeys!

Dear manager of the employee with Imposter Syndrome

Jean-Mark Wright — Tue, 12 Dec 2023 15:33:50 +0000

Originally posted at https://jaywhy13.hashnode.dev/dear-manager-of-the-employee-with-imposter-syndrome.

Introduction

I dedicate this article to every manager with a direct report suffering from Imposter Syndrome. It's a silent killer that suppresses and humbles. Like myself, many engineers have felt the crippling sting without the mental or emotional awareness required to label it. The corollary is that balance eludes us while paralysis erodes the progress that could have been. Through the scars, I've reflected on what I thought would help me with the hopes that I can help someone else navigate this maze. My message today is to the managers because I think you're uniquely postured to make an indelible impact on the trajectory of your employees... if you're careful.

My story

Hi. I'm a Jamaican-born software engineer. I spent my formative years in a small company in Jamaica. I started as the lone engineer until I built a team of about five developers. Being in a small company meant I wore many hats. I was the chief technical officer, chief architect, product manager, occasional designer, customer support hero, engineering manager, and individual contributor. I spent 8 years building many solutions from the ground up.

Of all the products we built, I'm most proud of two particular ones. First, we built an Incident Management and Reporting system for our government. It had forms to collect information on major crimes and fatal accidents and an interactive map that clustered the data points once you were beyond a certain resolution. Blah. The part that excited me was the reporting engine that we built. It did some fancy introspection of the database schema and exposed a really powerful interface that permitted the generation of tabular reports and charts along with filters to refine the output. The expressive power was unusually satisfying. We built a poor man's version of SPSS. Using the interface we built they could visualize and capture so many perspectives on the data collected.

The other project I hold dear is an Insurance platform we built from the ground up. Never mind that you could get a quote, purchase motor vehicle insurance and submit claims - the technical underpinnings were much more compelling. Using thoughtful component design, we built a premium calculation engine, notification system, and a bevy of other features on top of a small set of modular components. The emphasis being on the "small set" of components with high reusability. Reflecting on those days always brings me pride and joy.

The unfortunate truth, however, is when I started working in Canada, you would never have known I was remotely capable of building anything sizeable. Unless you saw my resume, you would never have thought I could be trusted with the autonomy to build a small component, much less, design an entire system. You would never have known I've conducted numerous trainings, addressed countless execs with confidence, written numerous technical specifications, and obsessed about customer experience.

When I migrated to Canada, it appeared my confidence and much of my self-esteem missed the flight. Whatever was left of my years of experience was likely surrendered at immigration. Somehow I convinced myself that all my experience was third-world, and by extension, subpar. All of the previous years I concluded were irrelevant for the years ahead of me. I had zero leverage, I'm here to learn from all the mighty folks who've worked with systems at a greater scale than I've ever imagined.

When I landed my first job, I joined a team that was pretty happy to have me. Never mind the warm welcome, deliberate inclusion attempts, or my manager's attempt to convince me that tangible contributions weren't expected yet, I was determined to hit the ground running flying. I probably deployed in the first week. I deprecated some unused code, and then quickly moved to more substantive changes. "He totally skipped onboarding" was a common rhetoric my teammates used to caption how quickly I integrated myself into the team.

The struggle to be relevant quite likely explained what was left of my voice and dominated my motivation. I managed to express my opinions, and I did in many team discussions. In retrospect, I wasn't remotely as confident as I was previously. I made several suggestions and hinted at improvements I never followed through on. I bowed out of several discussions where I swallowed my passion and conviction on the topic. From the early days, I saw opportunities where I could make an impact, but I convinced myself, my thoughts would be deemed primitive. I didn't want to risk exposing my third-world experience, or identity in a grown-up world. So, understandably, 1-on-1 most conversations about opportunities stayed right between my manager and me, where they'd be best kept safe. In the early years, there was very little kinetic that exposed the potential within.

There were a few other factors that compounded the problem. First, on starting this new job I was overwhelmed by all the unfamiliar tools they used on the job. The bevy of tools reinforced the message that I wasn't doing real engineering and that my learning had just begun. I didn't use GitHub or do code reviews, I wasn't accustomed to defending my position or articulating my design choices. We never shipped logs to a third-party tool or used a dedicated tool to capture errors, or used metrics to evaluate and monitor systems.

Secondly, an overdose of personnel changes paralyzed me or undid much of the progress I made. In the first couple of weeks, our designer was fired. He seemed pretty competent and I liked his work. I was eventually told that there were unresolved, long-standing issues that aggravated the team and negatively impacted progress. Not before I cross-examined myself trying to determine how I could be next. After that incident, my manager left some weeks after. In the second year, our company was acquired. This change inspired quite a bit of uncertainty across the company. Ultimately, many of the original teammates, and manager at the time left. The teammates who created the welcoming atmosphere left, along with the manager who I first had conversations about seniority with.

Thirdly, two persons eventually joined our team which made me particularly self-conscious about my competence. Two exceptionally competent engineers with really strong opinions joined my team. One was an IC, and the other was my manager. The former ran his own startup at one point, and the latter had names like Google and Amazon on his LinkedIn. This helped me to deepen my bond with my insecurities. In those days I learned to specialize in overthinking and moving just enough to escape the definition of immobility. I learned a lot from them both, but my concerns about appearing incompetent repeatedly crowded out opportunities to acknowledge the progress made to a shared outcome. I would focus much more on the debates I lost, and how I could avoid such ignorance in the future.

It took several years to build the perspectives that helped me retrospect on my pre-Canada years in a way that enabled me to see my own leverage. I learned to caption my previous experience, not as insufficient, but rather, as different. Everyone has different competencies, and we're all imbalanced in some respect. I realized the tools I never used before were all accelerants for my current capabilities. I discovered that articulation of technical design choices was a simple, practicable skill. I read and got a lot better at articulating the gut feelings that predated coming to Canada. I started adding more structure to 1-on-1's and deliberately sought feedback that helped calm my limbic system. I grew to love the input from teammates with stronger opinions and used them as a deliberate vehicle for critically evaluating my designs. I looked forward to and sought out code reviews from them. I started focusing on outcomes, rather than on individual positions (actual quote from the ex-lots-of-big-name-companies manager I had).

Most importantly, I discovered that leveraging some of the disregarded foundations of my formative years made an incredible platform for career growth. I've long been pretty passionate about education and sharing and building bridges between teammates. Seniority requires impact outside your team, forging bonds that deliver value. By reviving my passion for sharing and education, I've been able to use that talent as a vehicle for leading cross-team initiatives. Last year, for instance, I delivered training to some folks in Engineering challenging us to level up on how we monitor and observe our production systems. I'm currently training trainers to spread the learnings to the rest of the Engineering org, chipping away at the bigger vision of changing the culture to build more debuggable systems. There are also plans to consult with another team as they rebuild certain areas of their system with a renewed focus on Observability. It's thrilling... and a little terrifying!

Five years later, I'm a Senior Engineer II at my company, and proud of my progress thus far. I'm incredibly grateful to all the folks I've encountered along the journey. There are so many people who played a key role in getting me to this point. I haven't fully relinquished my citizenship as an imposter just yet, but I'm working on it. All in all, I believe I could have compressed a few years, but you don't get to keep the lessons and the impact, so I'm still trying to embrace the journey.

My advice

Know your employee

I think one critical point is the importance of knowing your employees. For starters, take their resume and have a conversation with them about their past work experiences and what they've worked on. Most folks can speak positively of past accomplishments. This will help to provide a baseline that provides perspectives that aren't always discovered in organic day-to-day interactions. Conversations about past work can help to uncover latent or otherwise undervalued skills. This also provides an invaluable reference that can be relied on later to determine how to best grow that employee. This is also a great opportunity to validate some of those experiences and explore ideas for growth. Having these conversations early before the team dynamic and other environmental factors crowd out the motivation can be pretty powerful and help accelerate or combat irrational fears.

Help them to practice objectivity

Recall when I moved to Canada, I convinced myself I had zero leverage. I downloaded my accomplishments and disregarded my past foundations. This proclivity only increased as the years passed by. I continued to view my achievements in a lesser light. Rather, actually, I never identified or gave them any spotlight. My new focus was a yardstick that shifted aggressively as I received feedback or insights from someone else.

What would have helped was external perspective unblinded by the irrationality that dominated my personal evaluation. I wanted someone to question the downplayment of my work. I needed someone to challenge the utility and impact I had.

By far, it's not a panacea, but challenging your employee helps them peel back the layers of irrationality that cripple their ambition and attenuates their progress.

Strongly encourage self-reviews

A powerful tool for building objectivity is self-reviews. You should really encourage your reports to do it. There's a guide here that was particularly helpful for my last promotion cycle and I wished I had been using it all along. In this article, the author shares her very thorough process for doing self-reviews. In short, you're cataloging all the work you've done over whatever period you're reflecting on. You start with your memory, then you check all the tools (JIRA, GitHub, Google Calendar for meetings, Notion, Slack) to catch things you've missed.

You end up with a compelling list that details your progress - a real dent to the irrationality that typically reigns. It subdues the voice that shouts imposter, replacing it with fine details of what you've actually accomplished. When I did the exercise I was particularly shocked at the countless initiatives my mind glossed over when quantifying my impact. If you have employees you're proud of, self-reviews will help them better see your perspective. As a bonus, if your employee does a self review this makes your job much simpler come promotion time. You'll be able to pick from the catalogue your report has already drafted.

Talk about energy levels and expectations

As a growing imposter, I would constantly keep my pace in check. I would keep trying to maintain an unreasonable pace where I could convince myself that I wasn't doing horribly. There were periods where I felt somewhat accomplished -- but those periods weren't void of the self-inflicted pressure to maintain that pace. It was a vicious and unforgiving cycle. I'd beat myself up for not doing enough to feel accomplished, and then I'd eventually feel accomplished, but then I'd burn out and start beating myself up again. What a cycle!

As a manager, help your employee understand that energy levels ebb and flow. There are times when we'll feel energetic and ready to conquer the world. However, as sweet as victory is, the battle that took us there is also quite draining. It's fine to relish in the sweetness of victory, while we rest and lick our wounds. Rephrased less poetically, it's okay to grab some mindless, smaller tickets after you've just landed a huge PR. Or, after finishing a round of technical design where you refined your approach with feedback from your colleagues, it's okay to take something less intimidating.

Level with your employee! Help them understand that you experience these struggles too. We all do! I appreciated my manager responding to some of my questions about feeling unproductive. He was transparent enough to share that he felt very unproductive at points and shared some perspectives that helped. Often it's just our minds deemphasizing the impact and focusing on impostership. You can help by being transparent about productivity and energy levels.

Understand the power of feedback

I close with a petition to give your employee thoughtful and thorough feedback. Your feedback is often the only compass in the conversation that has the potential for accuracy. The internal compass of a budding imposter is deeply skewed. I don't know that compasses are repaired, but loaning your compass can create the environment for our compasses to heal.

Give feedback regularly, and strategically. Look out for the peaks and troughs for your employee and give feedback targeted around those points. When they've shipped a massive new feature, be prepared to engage and offer critical feedback. Find out how they felt about it. That's a unique opportunity to provide perspective about the impact their compass has crowded out, or that they aren't privy to. Also a great time to preempt conversations about rest.

On the flip side... when your employee is going through a low phase, that's also a great opportunity to connect and understand what's going on. Having open conversations can lead to a formalization of a plan that facilitates their rest, or whatever else they're going through. For example, let's work on smaller tickets for the rest of this month and then we can check on your energy levels. A formal plan with the manager really helps to silence the inner irrationality flogging you to keep going.

Your feedback, if you're careful, can really being the transformative process that results in less aspirations for impostership.

Conclusion

In conclusion, if you wear the coveted title of "Manager of the employee with Imposter Syndrome", remember to whom much is given, much is expected. You've got to really understand and nurture your employees. By initiating conversations about past experiences, managers can unearth hidden skills and provide a platform for growth. Encouraging self-reviews and objectivity is essential to combat the pervasive influence of irrational self-doubt. Recognizing the natural fluctuations in energy levels helps employees maintain a healthy and sustainable pace, fostering a more authentic and productive work environment. Crucially, regular and thoughtful feedback serves as a guiding compass, offering perspective and support during both triumphs and challenges. With these strategies, I'm hopeful that you can help your employees steer clear of the pitfalls of imposter syndrome.

Thanks for listening!

How I keep my work life organized with Notion

Jean-Mark Wright — Fri, 30 Dec 2022 04:42:50 +0000

This article was originally posted here.

Introduction

About two years ago, I broke up with Evernote and made the switch to Notion, and I haven't looked back since. I use Notion to keep me organized at work. By organized I mean... keeping track of everything I need to do, having a suitable space for collecting and refining thoughts, chronicling lessons learned, and immortalizing meetings.

Evernote - the breakup

For many years, Evernote and I were going steady. I used Evernote to capture meeting notes, capture my thoughts, be a scratchpad, and do lots of other things. We started to have problems when I realized I was spending too much time trying to make my notes in Evernote look attractive. Yes, apparently an attractive writing environment accelerates my productivity. Countless studies I've never done establish this fact. I was trying too hard. Evernote had too many controls and knobs and I was growing wary of fiddling with them and failing to achieve the aesthetic I was in search of. Lots of choices for font, size, color, heading style and alignment, etc... It was a dizzying array of choices. By default, things were quite bland, and it took a lot of work, too much actually to make it look better.

In came Notion

At work, a developer had put together the results of some research in Notion. I was intrigued. When I checked it out I realized Notion has some interesting features that were pretty compelling. Notion has a concept called pages. A page is basically a simple document that supports Markdown. I could create documents with different headings, bullet points, checklists, quotes, code blocks, images, links, and various embeds. Also, it had great support for embedding emojis within documents, and you could add cover images to all your pages. Best of all, it only had only three font choices, all of which looked great. It meant, by default I had the aesthetic I was in search of. The writing environment looked great without me having to try. I could easily add cover images from Unsplash to my pages easily through their image tool. With little to no effort, I could make a page feel so inviting and so inspiring. It's just what I wanted.

Finally, what sealed the deal for me was Notion's concept of databases. In Notion, you can create a database and visualize them in different ways (table, calendar, card view). Additionally, you can create different views of the same database and apply filters and sorting. And it all looks great! No need to tweak any aesthetics.

When I discovered databases, I also stumbled across another feature I was just trying to do manually with Evernote. Templates! With Notion, I can define templates for each database. Meaning, when I'm adding something to the database, I could use a template to capture common elements. This feature turned out to be super handy and I currently use it a lot!

The specifics

Enough with the elevator pitch! Let's talk about how I actually use Notion.

I use Notion for meeting notes, todos, chronicling lessons learned, my scratchpad, and for organizing large chunks of work.

Meeting Notes

I have a Meeting Notes database in Notion that I use to plan, and capture notes and action items from meetings. I created a few fields to keep on top of my meetings.

Date - capturing the date of the meeting allows me to show sorted views of my meetings. I was able to create a view for Upcoming Meetings. This was handy for quickly isolating meetings where I'm planning things in advance and want to capture notes in the agenda.
Tags - I added tags to help me logically group meetings. Currently, I use tags for different projects. When I worked on Multi-factor Authentication, applying that tag helped me group all my meetings under one umbrella. Also, since Notion allows me to use different views of the same database, I was able to list all the MFA meetings on a page I created for MFA.
Reviewed - I recently started aiming to review past meetings. I haven't done it religiously, but I added a boolean to keep track of which meetings I had reviewed. This allowed me to create a view for "Meetings to be reviewed".

I also use a template with my Meeting Notes database.

The template has the following sections.

Agenda - I use this section to plan for my meetings in advance. I generally capture in bullet points what we're going to discuss in the meeting.
Notes - I use this section to capture free-form notes as the meeting progresses. I generally use bullet points.
Summary - after each important meeting I generally try to summarize the meeting. The goal is to synthesize the points into a cohesive narrative that can be shared with all the participants. I've found this is a great forcing function to stay attentive in meetings and for achieving clarity. I use it for all important meetings I lead.
Action Items - I capture any actionable items that come up during the meeting or as I'm reviewing the meeting notes.

Two years and hundreds of meetings later I almost feel like Sherlock. In the past two years at work, I've led major initiatives, and having these meeting notes has been incredibly helpful. I can always time travel to revisit conversations and reflect on the different perspectives. Since I have so many archaeological artifacts I can always confidently build and extend what we actually discussed, instead of relying on my best memory of these interactions. How do you capture your meeting notes?

Organizing chunks of work

As a Software Engineer, I'm typically working on some deliverables as part of my daily function. These deliverables vary in size. I may be working on a ticket for my team that takes anywhere from a couple of hours to a couple of days. This could be feature work, bug fixes, system enhancements, or system design. Other times I may be designing a bigger feature that will eventually result in smaller tickets. I use Notion to help add structure for both.

For Tickets, I have a template I use that reminds me to think critically about the work that I'm doing.

Let's talk about the template sections:

Summary - In this section I remind myself to capture my understanding of the ticket. This forces me to think critically about the goal and deliverables for the ticket. I have a placeholder that reminds me to describe why we're doing the work as well. This section has highlighted misunderstandings and points for clarification.
Links - here I capture any relevant links I need for the scope of the ticket. For starters, I link to the ticket and any pull requests for the work I'm doing.
Todos - I use this section to capture my action plan which typically consists of several tasks. I can order them, use indentation to create subtasks, and then check them off as I go along, it's a simple section that reminds me to get the to-do list out of my head and down on paper. It's much more effective having the tasks written down. There's much less cognitive weight. As a quick example, a task may require that I contribute code to several repositories, or sequence my delivery in increments. Especially when the work spans multiple days (which it mostly does), it's great to have the context written down so I don't have to reconstruct my progress each day.
Scratchpad - I use the scratchpad as another way to help me organize and process my thoughts. I use this section to think out loud. Thinking out loud on paper is much more conducive to analysis and retention. Often I go through many cycles and tangents while I'm thinking. Writing it all down does a much better job of facilitating all the journeys I want to traverse to explore and articulate a problem.
Resources - I use this section to capture links that aided research and completion. Performing a task often requires a bit of research. There's always something new to learn or something I can't remember. Linking to helpful resources builds a strong set of references I can revisit when I'm in need of a similar solution.
Lessons - this section is a reminder to reflect on what I've learned as I did the task. I'll speak to this more directly later.

Often I'm working on larger chunks of work that will eventually translate to smaller tickets. This could be an entire feature I'm ideating or research related to a bigger feature. Recently, for example, I was looking at how we would authorize calls between one of our core microservices and a new one. These tasks require lots of thinking. Two quick examples of deliverables are design documents comparing alternatives, or an architecture brief providing an overview of a particular design we've chosen. This calls for a different structure from daily tasks, so I use a different template for these tasks.

Here are the sections.

Links - in this section I capture links to relevant documents and items that I'd waste time trying to find typically. If I've got a design document in Confluence I'll link it here. If there's a design done in Figma by our design team, I'll capture it here. I also link to the JIRA ticket that captures the scope of work.
Todos - the intention for this section is pretty similar to tickets. Here I'm breaking down the work I'm doing and ensuring I capture the different steps I want to take. This section is usually grouped into broad headings. I typically collect action items from meeting notes in this section. It helps to have everything in one place so I have an overview of everything that needs to be completed. If there are broad themes or other logical headings, I leverage those to group the tasks I have. I'll also mention that the task list also comes in handy when I find there's a tangent I need to explore later. It allows me to do a brain dump so I don't get distracted from my current direction of thought, while still allowing myself to revisit the tangent and explore it.
Meeting Notes - in this section, I use a filtered view of my meeting notes database to capture links to meetings on the topic. I use meeting tags that allow me to group-related meetings. The view lists my most recent meetings first. It allows me to quickly jump to past meetings to review perspectives, ideas or specific points.
Resources - I'm using this section in a very similar manner as for tickets. I'm capturing any resources that helped me think more deeply about the task or provided some breakthrough in my thinking approach.
Scratchpad - this section is exceptionally critical to help me capture and refine my thoughts. When I'm working on larger tasks there is a lot to explore, analyze and contrast. In this section, I'm literally thinking out loud. As a quick example, I may be thinking about the pros and cons of one design. Instead of thinking in silence, I vocalize it by typing it out. It helps me to be much more coherent and thorough in my analysis. Notion also has a MermaidJS integration which allows me to capture sequence diagrams (it also supports other diagram types). I like to use sequence diagrams to capture system interactions for APIs. These tools allow me to iterate and vet different approaches or ideas quickly. It's just what I need when I'm working on big chunks of work. It can be a little overwhelming to keep it all in your head. My scratchpad provides a great place to try out ideas, explore, and refine them.
Lessons - I've definitely found that there's so much that I learn in working through bigger ticket items. Bigger ticket items are a melting pot for growth, and I like to be deliberate about capturing learnings so I can consciously apply them in other applicable contexts. I've learned a lot from manager 1-on-1's, meetings on the topic, and comments on documents I've written. I love hearing how other folks think and approach the topic or area I'm working on. It helps me to augment my perspectives and concretize my own thoughts on the topic.

Lessons

One principle I'm always guided by is "anything can be hard, but only the first time". A close second is to foster a dislike for repeat problems. I believe our characters and our minds should be controlled models of dynamism. We should be deliberate about growth! One sure way to stimulate growth is to reflect on experiences and extract learnings from them.

I have a database in Notion, just for that! I aim to capture what I've learned by associating them with a phrase or proverb. The key idea behind using proverbs is that it breaks problems into their fundamental parts which are more easily compared to other seemingly distant problems. This helps bring existing solutions in reach for a seemingly novel problem.

As a quick example, I wrote "too many cooks spoil the broth" when I was reflecting on some APIs I designed that carried too many responsibilities. It became a reminder to decouple intentionally to ensure I build lean, focused, and maintainable APIs. I've since returned to the same proverb and added a lesson I was reading from an article on Shopify's Dev Blog. They told the story of how they deprecated an entire system that cached events for a front-end map and ended up with lower latency overall. Now the next time I'm working on something that feels a little too weighty, once I've articulated the problem simply, I can now easily access other solutions and apply them.

Finally, I used Notion's Card View and I added images to associate a picture with the lesson. The images promote stronger retention of the proverbs and their associated lessons. The images are another channel whereby I can connect the dots between one problem I'm looking at and others I've seen previously.

Conclusion

Thus ends the story of how I keep my work organized with Notion. By using Notion's templates and databases I'm able to keep on top of things. I keep track of meetings by capturing the essence and extracting lessons I want to remember. I think out loud using Notion's simple writing tools and diagrams to promote greater clarity and comprehension for my daily tasks. Then, I have a dedicated area where I reflect and highlight important lessons I don't want to forget. All these tools have been invaluable in helping me be productive at work.

I'd love to hear your story! What tools and practices do you employ to inspire productivity at work?

Can a good explanation really prevent a prod incident?

Jean-Mark Wright — Tue, 13 Dec 2022 03:35:54 +0000

Originally published at https://jaywhy13.hashnode.dev/can-a-good-explanation-really-prevent-a-prod-incident.

A missed opportunity to communicate

It was March 21, 2022. I remember the day like yesterday. I just started my On-Call Shift. Our team On-Call rotation is set up so each engineer goes on call for a week, once every 6 or so weeks. The On-Call Engineer is responsible for responding to incoming customer support requests, upgrading packages for our core microservice, and addressing a dedicated backlog of small system enhancements. Our core microservice is one of the older systems in our company. In the earlier years, priorities, unfortunately, trumped the kind of TLC necessary to keep the microservice lean and focused. Our microservice boasts lots of outdated packages, and accreted impostors that never got evicted. Despite successful efforts to rectify this, there still remains an entire journey ahead. As such, the On-Call is never bored, or without a steady stream of work.

I walked into this shift feeling inspired, and desirous of making material improvements. Having worked with the same service for 4 years, I carry the burden of service improvement every day. Unsettled with the monotone of package updates, I sifted through the list of updates looking for an opportunity to make a bigger impact. As I scanned the list of packages that required updates and stumbled across two third-party packages I didn't recognize. What's defusedxml and bcrypt?!? Why do we need those?

Let's start with defusedxml, I thought! I use VS Code for editing, despite my unresolved feelings for Sublime Text. I opened up the file with the list of packages and their versions. With one keystroke, I invoked a nifty feature of Git Lens, which deposited the Github URL for the commit to my clipboard. I quickly dropped the link in the browser to start the interrogation. I was anxious to see the commit message...

"saml2 backend for [company X]"

Hmmm... not too much context there. Somebody was trying to add a SAML authentication backend to our Identity microservice. SAML is a Single-Sign-On mechanism. Some years ago, we wanted users from Company X to be able to sign in to our Identity microservice once they had an account at Company X. Ok... but what does that have to do with defusedxml? Maybe let's look at the PR. I found a PR with 18 files and ~1000 lines of code. The author had long left the company years ago. I did some additional research on the package and discovered it's used to prevent certain types of XML attacks when parsing XML. There were no direct references to the package in our code. I concluded that there was perhaps some interest in using the package, but the engineers never got around to using it. Seemed safe to remove!

Now for bcrypt! I checked across our codebase and realized there were no direct references to the package except some old, unused SQL scripts that were used previously to move data from prod to our staging environment. Enough for me! Let's remove the packages and get a PR out there!

I baked up a quick PR removing the packages and pushed my branch to Circle CI for testing. In parallel, I also checked a few endpoints to ensure nothing critical was broken. The tests in Circle CI were all green! I wrote a descriptive commit message.
I captioned my commit message, then provided my rationale in the body of the message.

bcrypt - this is a package that provides some crypto functions for hashing. It was never used across our repo. I'm not sure if this was a dependency of the Django password hasher previously. However, today, we're not using it.

defusedxml - this package is apparently used to prevent certain types of XML attacks when parsing XML. It was added in this PR while the Company X integration was being developed, but it doesn't seem to have been used. We still have Company X code to remove but none of it is being used today.

I got the +1 to merge, and even a heart emoji from my manager! Who doesn't love to see unused code being removed?

As my finger depressed the mouse over the merge button, there was a sense of pride welling up inside. I felt like a proud citizen of our team, and steward of our microservice. I had discovered defunct dependencies and chipped away at the unnecessary cruft inside our service. For all intents and purposes, I could have ended my shift early! My work was done!

Not so fast

But then, Slack started getting noisy -- one of our monitors went off. People were having issues logging into our site. The monitor was quickly followed by a trove of Sentry errors -- something about bcyrpt missing. It turns out, bcrypt was actually required.

Wait... what?!?

I could feel the contentment being collected by the alerts. I saw the alerts, and was able to jump in and roll back the deployment. The incident lasted just a few minutes, and not too many customers were impacted. Hmmm... I guess bcrypt is actually a thing, huh?

After deeper investigation, I realized that bcrypt is actually a runtime dependency of our system. We use Django, a Python framework that provides lots of batteries. One such battery is authentication. Django provides functions to authenticate users with some configurations. One such configuration is the password hasher. We dictate the hashing algorithm Django uses when persisting and verifying passwords. To complicate things, apparently, bcrypt is rather slow, so we disable it in Circle CI. This explained why all our tests passed. None of them ran using this hasher.

Thankfully, nothing actually broke in prod related to defusedxml. Seems like I wasn't incorrect in considering defusedxml safe to remove. However, I rolled back the change so we could perform a deeper investigation.

What did I learn?

So let's talk about what I learned from this incident.

A good commit message goes a long way

Writing a good commit message is really critical. You can use a commit message to help future you understand why you introduced a change. Usually, someone can derive what was done by looking at the change, but the why is that much more important! Years later, someone may be looking at your code with the thought of modifying it or removing it. How will someone know if your code can be safely modified? A good commit message bakes in the context for why the change was made, and all the working assumptions involved in the introduced change. More importantly, we can illuminate any potential conundrums. However, commits aren't the only place where we can provide context.

Context can (and should) be provided in many forms

In the case of the bcrypt dependency, it would have really helped to have a comment right beside the dependency (in the dependency file) explaining why it was added. Additionally, that would also have been a great place to help folks understand why bcrypt wasn't used in testing. Those are some inconspicuous connections that are worth highlighting for folks. I realized after that I didn't review the commit history for bcrypt. I must have been riding the adrenaline rush after discovering defusedxml was unused. Sure, I could have done more digging. However, that's a personal change we can't guarantee other engineers will use. Everyone has their own approach and may check different places. The more places the context exists, the better! The dependency was quite obscure. It took a fair amount of digging to realize the connection between bcrypt and our authentication system.

This other time a great commit saved prod

Fast forward to another day when I was working on a ticket to upgrade an outdated package that was pinned. By pinned package I mean we've told our package manager that we only want a specific version for a particular package. Some of these packages end up being pinned for years and never get updated. Ideally, packages are unpinned and the lock file captures the actual version. However, this time, we had an oauth2client dependency that was pinned, wasn't updated, and was now replaced by a new package Google had.

In this case, there were some dependencies that were pinned whose version conflicted with the new package Google came out with. I couldn't install Google's new package without upgrading those dependencies. I wanted to know if I could safely upgrade them. Were they pinned for a particular reason? Was it safe to upgrade?

Thankfully, when I took a look at the commit history I found a fantastic explanation that indicated that the packages were pinned to avoid breaking changes in yet another package. So, I was trying to replace oauth2client, but I needed to upgrade package X, and the commit told me that package X's version was pinned to avoid a conflict with package Y, which happened to be X's dependency. It was just a tad convoluted, but I was eternally grateful. Without this commit message, it meant I could have updated package X and then broke prod because of some conflicts with package Y.

Conclusion

Don't ever think a little extra can't prevent some downtime in production. A great commit message can provide a lot of context and really give us the confidence to make modifications. Remember though, it's not just commit messages! Think about how you can sprinkle context in other helpful locations so we help others and our future selves avoid making destructive changes!

Do you have a story of how commits prevented a disaster, or other ideas to preserve context? Let me know in the comments!

You don't always need a queue

Jean-Mark Wright — Thu, 08 Dec 2022 11:53:08 +0000

Originally posted at https://jaywhy13.hashnode.dev/you-dont-always-need-a-queue.

Introduction

A few weeks ago my colleague and I were talking about the pros of using a queue to take some work out of the request cycle. He read an article that suggested that queues should be used for all external calls. I disagreed. Making broad recommendations like that generally don't pan out because there are always tradeoffs. It's bound to be much nuanced than that. In this article I want to explore a few considerations worth exploring before you jump to using a queue.

Pushing work off to a queue isn't free

The first thing worth considering is that pushing work off to some queue isn't without implications. There are many operational considerations. Firstly, how long does it actually take to push the work on to a queue? In the best case, it takes less time than the actual work you're trying to remove from the request cycle. However, that might not always be the case. It's possible that the call you're aiming to remove from the request has comparable latency. Say it takes 10ms to enqueue the job but it takes 20ms to make your API call, is it really worth pushing that work off to a queue?

Next, we need to remember that queues aren't faultless mechanisms for getting work done. Queues have operational issues we also need to cater to. The workers that service requests could be inundated, there could be issues with the broker responsible for relaying messages, or other issues that cause delays or other problems. We'll need to monitor the work done by our workers, implement sufficient retries and ensure we have mechanisms to process work that was unsuccessful (e.g. by using a dead letter queue). By pushing work out of the request to an entirely separate mechanism, we will inherit some operational overhead we need to cater to.

Consider the total latency

As a follow on from the previous point, it's worth considering the total cost. The main benefit of removing work from the request cycle is to provide a better experience for our customers. Ideally, we don't want our customers waiting if they don't need to. Using a queue doesn't necessarily automatically give us the time savings we're in search of. The only way to confirm we're getting the savings is to examine it!

How long does it take to enqueue a request to the broker? What does the latency distributions look like? What's the p99, p95, p90, etc... Does the queue have an uninteresting latency distribution? By uninteresting, I mean it's a unimodal distribution with a short tail. In other words, ideally it has one thin peak with little variance. The peak tells you the most likely range of latency you should expect. A thin peak (low variance) tells you the latency won't vary too far from that peak. The tail length indicates the upper bound of the latency. A shorter tail should mean requests shouldn't take too long in the worst case. You want your queue to be as boring as possible. It should be really quick to enqueue requests and the time to enqueue shouldn't vary significantly between different requests. You also want to consider the time it takes for a worker to pick up the request. You want workers to pick up requests quickly. Finally, how long does the worker actually take to do the work?

Now consider the latency distribution of the API call you're thinking of performing asynchronously. What does the latency distribution look like for that? In the worst case, it'll have an entirely different latency profile. It might not be as optimized as your queue, but you should confirm it! In the best case, it could be just as boring and unremarkable. If it is just as performant or more than enqueuing and servicing your request, then you haven't saved any time!

So in essence, how long does everything actually take? If it's negligible, it's quite possible a queue only provides additional overhead without considerably improving customer experience. If it does take long, quite likely the additional overhead is going to be necessary to provide a palatable customer experience.

Queues break the feedback cycle

Another very important consideration is that using queues breaks the organic feedback cycle afforded by the original request. When a request comes in from a customer's browser, we can leverage that request to provide feedback. Whether that feedback is informational or exceptional, the request provides a channel of communicating with the customer. Once we push work to a queue we need to establish an alternate mechanism of communicating with the customer.

It's important to note that feedback can take different forms. Sometimes we only need to provide unactionable feedback like "we're working on it!", however, there are other circumstances where we need to communicate with the customer. There could be a validation problem, or some other failure or exceptional problem. If the original request has been completed how do we communicate with the cuatomer?

Let's say you enqueue a request to send an email (by calling some external provider like MailGun). What happens if MailGun returns an error. How do you notify the customer of the failure? You'll likely need some additional APIs to provide the feedback cycle the original request afforded you. Whether that takes the form of additional REST, GraphQL or even socket based. You'll need some other way of establishing a communication channel.

Conclusion

You don't always need a queue! If the work you're offloading doesn't require significant time, a queue might be overkill. Don't assume always using a queue is what you want to do. Consider the operational costs of running a queue, look at the total latency savings, and ensure you have a formidable feedback mechanism for communicating with the customer.