DEV Community: Alexey Antipov

Observability & Source Maps: Through thorns to the stars

Alexey Antipov — Wed, 14 Feb 2024 14:01:47 +0000

TLDR: Sentry has come up with a simple and reliable way to associate production code with source maps and set it working. No excuses for not using source maps anymore.

Around this time a year ago I was busy finding my way around a proper observability setup for Flink’s (my employer at that time) microfrontends. The main hurdle was dealing with source maps in Datadog (an observability tool) which hadn’t supported microfrontends natively and I was looking for a workaround.

Why source maps?

Code errors reported to an observability service (e.g. Sentry, Datadog) contain stack traces with a chain of functions calls which lead to the exception. This information includes function name, file name and place in the file where the function is called. That should help developers in debugging their code - locate the problematic code faster.

The problem though is that our code in production is obfuscated and minified. This means that function names, filenames and function calls locations from stack traces are of no use to us.

To solve that problem observability services use source maps - kind of a mapping between obfuscated production code and raw code. Production errors’ stack traces get translated into true function names and their calls locations in the original raw code. Observability tools even restore the raw code in the vicinity of each function call. As a result, pinpointing the problematic code becomes much easier.

Where to get source maps?

Source maps are files with .js.map extension. They are generated by code bundlers e.g. Webpack, Vite. For each [filename].js file an accompanying [filename].js.map file is generated.

When deploying your code to production, you need to upload the associated source maps to your observability service.

The problem with source maps

When dealing with source maps, the hurdle lies in association of minified [filename].js files with their [filename].js.map source maps. Depending on your setup, filenames may persist but their contents change with every deploy. Your observability service end up having multiple source maps for a single filename. Which one should it use to deobfuscate an error stacktrace at hand?

To overcome this issue, most observability tools rely on extra information: releases (versions). It requires you to perform two extra steps:

for each production deploy, you should generate a new release number and provide that number when uploading source maps to your observability tool. One release number for all the files per deploy.
In runtime, when initialising observability service, you tell what release number to use. That information will accompany all reported runtime errors.

Observability services take the release number from the reported errors and use it to look up for the appropriate source map file associated with it.

That approach is not a silver bullet though and is not 100% reliable because applications in production might use code from different releases. Imagine a long lived large SPA applications with code splitting: a user loads one page with a set of scripts in the morning and another page with another set of scripts in the evening. Two sets of scripts might belong to two different releases, but the running application keeps the same release number since the initialisation time. That means that errors for later loaded scripts might have wrongly associated source maps.

Beside that problem with reliability, such approach also doesn’t fit the use case of microfrontends. Each team deploys their microfrontend independently and has their our own releases. What release should the final application use when initialised? One can come up with workarounds (that is what I was doing), but that will just add more to the overall complexity around the source maps handling and make the reliability problem even worse.

Debug IDs to the Rescue!

Sentry has come up with a really neat and elegant solution to the above problems.
Instead of generating releases and assigning them when uploading source-maps and initialising the application, one simply includes an additional post-bundling step - running the command

sentry-cli sourcemaps inject ./dist

which will generate a unique debug-id identifier to each script in the output ./dist folder and include it in the script files as well as into the accompanying source map files. And then proceed with normal deployment and source maps upload.

With that, every stack frame will include debug-id in addition to function name, filename, and location. It will be used to associate a particular stack frame (i.e. function call) with a particular source map. That solves the reliability problems: all the files self-contain the necessary for the association information. No need for dances around releases.

(Source: sentry.io)

If you’re using Sentry and have either used the releases approach or haven’t set up the source maps yet, I encourage checking out the new approach. It has never been easier to use source maps with observability tools.

Not all tools support that debug-id approach yet. I checked Datadog docs and they still suggest using releases.

Hopefully situation will change soon and all other observability services adopt similar approach.

Cloudflare Workers: Unpacking My Fundamental Misunderstanding

Alexey Antipov — Fri, 09 Feb 2024 13:49:09 +0000

I’ve fundamentally misunderstood how Cloudflare Workers work.

While setting up Sentry for my Notion<=>Google Tasks integration app, I noticed that a single Issue in Sentry contained data (”breadcrumbs”) from different requests to the Cloudflare Worker. This was a startling discovery indicating a data leak between Workers requests.

That implied either of two things:

Cloudflare has a bug in their “beta” Queues Workers product
My fundamental knowledge and assumptions about Cloudflare Workers have been wrong for years since their introduction.

After a brief conversation in their Discord channel and with the Director of Product for Cloudflare Workers, I learnt that my understanding had been wrong.

I had believed that every Worker invocation is executed in a completely isolated environment, with a new V8 isolate created for each call and discarded right after the call is finished. This assumption led me to think that there was no need to worry about possible data leaks between calls, a problem inherent in most (all?) serverless platforms. This feature was one of the primary reasons I was excited about Workers.

However, the reality is that Workers isolates are actually shared between calls, which explains why a single exception in Sentry contained data from multiple calls.

Where did I get such misunderstanding from?

Part of the blame lies with Cloudflare’s How Workers works page, which, up to this day, has contained a wrong statement:

“Workers processes are able to run essentially limitless scripts with almost no individual overhead by creating an isolate for each Workers function call”.

I have created an issue in their docs repository to address that misleading statement.

The other part of the blame is on me for not thoroughly reading that page from top to bottom, failing to spot the contradiction and not validating my assumptions.

That revelation makes me to rethink my approach to building Workers apps.

A Tribute to React

Alexey Antipov — Mon, 29 Jan 2024 15:14:16 +0000

Attention: This article may cause discomfort, particularly for individuals not from the React community. No offense is intended. Proceed at your own risk.

Preface. There was a surge in criticism towards React over the past year. Complaints ranged from reluctance of React's core members to adopt the trending Reactivity API, to confusion over React Server Components. IMO, the significance of the issues is often exaggerated while the objective proposition value of React is neglected or diminished. In this article, I want to highlight one particular aspect of React that is hard to overestimate but often overlooked and taken for granted.

Ask yourself:

What has been the main pain point of frontend development in the past decade?

For many, it's the ever-changing landscape of frontend development and the need to constantly re-learn, rewrite and fight with broken npm updates. It has become normal for a hyped and trendy technology to fall out of fashion within 2-3 years, get abandoned and become a tech debt burden for projects that relied on it.

We are lucky if our use of the guilty library is limited and the solution is straightforward. But dealing with more complex situations can become a nightmare causing the notorious fatigue. Many of us bear the scars from migrating AngularJS and Vue apps when those frameworks couldn't find a way for further gradual evolution and seamless updates. Those unlucky enough to have bet on these frameworks felt the impact: businesses, developers, library authors.

Is there any major frontend technology that has stayed relevant for over a decade, never broken its ecosystem, evolved gradually, and continued growing in popularity? Is there such a bastion in this volatile field?

The obvious answer is React. It's a true phenomenon.

React has earned a reputation as the least risky technology choice for frontend development. It has so far been the safest bet for business owners, developers and library authors.

With huge momentum, React continues to evolve, and its ecosystem thrives. IMO, the apprehensions about React's state of affairs are exaggerated.

Why did React make it where many others fell short? Perhaps it is the initial ingenious design principles of React, that helped it stay flexible and evolve gradually. Or maybe it is the React team's ingenious, pragmatic and cautious architectural decisions along the way. Likely, it's a combination of both.

Let's appreciate React for being such a phenomenon.

Let's appreciate the immense achievement of React's team.

The team has truly deserved better treatment, more respect, more trust and more patience from us.

Be nice. Be cute.

THANK YOU REACT! LONG LIVE REACT! 🤘

Beyond Frontend vs Backend: The Case for Fullstack Development

Alexey Antipov — Wed, 24 Jan 2024 15:07:00 +0000

We are accustomed to the idea of distinct roles for Backend and Frontend development.

I used to like the idea and never undermined it; Fullstack development never appealed to me and I never believed in its effectiveness. Through most of my career I advocated for specialisation and perfection in a specific area. I chose Frontend Development to put all my efforts into.

Recently, though, I started getting increasingly convinced that such specialisation often does more harm than benefits to both parties - businesses and developers.

Why?

Firstly, that conviction has grown out of frustration of being a highly skilled and specialised Frontend developer. Deep knowledge and skills definitely helped me a lot in passing many interviews and acquiring confidence. But I rarely had a chance to make use of them at work. Most companies need just basic Frontend skills, they want you to build basic UIs, implement form validation, and apply component libraries. They don’t care about how readable, performant, maintainable and scalable code you write. They care only about the end result - delivering features. Honing frontend skills and perfecting in the ever-changing Frontend landscape was apparently not a good investment of my time and efforts.

Secondly, I’ve got to learn that some (most?) companies are really bad at managing their human resources and work load. I’ve seen many situations when frontend devs were out of work and were tasked to come up with some useless work to make themselves busy or had to learn and switch to backend development for a while until they have some frontend tasks. Examples of backend folks having too much on their plate were common as well. Both types of situations led to frustration and/or developers’ overwork. And I can see Why such situations are common - having separate Frontend and Backend developer roles means a lot of extra work of setting up proper communication and handover processes, separate interview and performance review processes, just to mention a few. There is simply too much overhead.

The trigger for changing my mind, though, was a story of ArsDigita company and its cofounder Philip Greensup that I read in the “Founders at work” book by Jessica Livingston. The company had a quite unique way of treating developers, utilising and growing their skills. Here are some quotes to give you an idea:

we tried to help each programmer develop an independent, professional reputation. We had this idea that programmers could be professionals, like doctors or lawyers, and, to that end, we wanted the programmers to be real engineers - to sit down face to face with the customer, find out what was needed … and then take a lot of responsibility for making it happen.

We pushed the profit-and-loss responsibility down to individual teams. For example, if there were two or three programmers working for HP, then those guys would be solely responsible for the project and making sure that it got delivered on time and that the customer was happy … Implicit in that was that, if it didn’t go well, we’d know whom to blame.

For programmers, I had a vision … that I didn’t like the way that programmer careers turned out. Now that I fly airplanes, I realize that the average programmer is really much less happy in his/her job than the average airplane mechanic… For $30,000 and a year and a half, you can become an airplane mechanic. You work in a small group, you meet the customer directly. You don’t have the alienation from the customer….

Airplane mechanics have a direct interaction with the customer. A lot of jobs require two to three people, so it’s kind of social, and I noticed that they’re just really happy. Programmers are isolated. They sit in their cubicle; they don’t think about the larger picture. To my mind, a programmer is not an engineer… If you look at civil engineers, architects, they’re all dealing directly with the customer and going through the whole process.

The programmers were in the corner doing what they were told. That’s one reason they were so easy to outsource. If a programmer really never talks to the customer, never thinks, just solves puzzles, well, that’s a perfect candidate for something to offshore.

You never really know what most programmers have accomplished. There are a handful of people that you can say that about. Linus Torvalds built the Linux kernel, but it’s hard to say what the average programmer working at a big company has ever accomplished… the projects are so big and their contributions were so small.

All that reasoning and observation lead me to change my mind. I realise now what most companies really need are fullstack generalists, rather than specialists. They need developers who are responsible for the whole cycle of product development, end-to-end. It’s much easier, cheaper and more efficient to manage a small team of generalists than a bunch of specialists. And developers, having and exercising that greater responsibility would be motivated better to deliver high quality results and would have more opportunities for creative work and clever solutions. It would be much easier for them to prove their accomplishments and grow professionally. And their job would be more satisfying.

Handling JSON data in SQL Databases

Alexey Antipov — Wed, 17 Jan 2024 15:05:06 +0000

I was implementing error handling in my Notion-Gtasks Sync app and I needed to store 3 additional data items in the database:

last error message
number of consecutive errors
next retry time

The idea was to implement exponential backoff for synchronisation attempts, limit the total number of attempts to 10 and decrease the number of duplicate errors reported to Sentry (I’m using a free plan). Therefore I had the following requirements for my error data handling:

be able to filter out rows with more than 10 errors
be able to reset all the error related data on successful sync
increment the error count upon encountering a new error

I’m an amateur SQL engineer and I decided to fulfil data requirements by creating 3 additional columns in the users table:

sync_error_msg
sync_error_num
sync_error_next_retry_time

It seemed straightforward, but something didn’t feel right. The thing is that all three fields are, by nature, tightly coupled - when setting, resetting or updating one of them, the others should follow suit. Placing the fields in separate columns breaks their ties at storage level, and we must rely on the code logic exclusively to maintain their relationships. It didn’t seem like a reliable solution; instead, it seemed error-prone. As the code and database schema evolve over time, it can be challenging to maintain those ties. Something more reliable and less error-prone was needed. Something like encapsulation, maybe?

Reading through my database provider’s docs (Cloudflare D1) I found a page describing sql functions to handle JSON data (based on SQLite JSON extension). It seemed perfect for my data handling requirements, allowing me to maintain relationships between the fields. I could store all my sync error-related data in a single column as a JSON object, as a single chunk of data.

I validated the idea and it yielded positive results!

Here are some of the implementation details.

I created a single column sync_error to store a json object with the error-related data.

// my sql table schema definition using drizzle-orm
export const users = sqliteTable('users', {
  ...
 syncError: text('sync_error', { mode: 'json' }).$type<{
  msg: string;
  num: number; // Number of consecutive sync errors
  nextRetry: number | null; // Timestamp in ms. Exponential backoff. Null if no retries left. Max 10 retries within 5 days
 }>(), // Last sync error data. Reset to null on successful sync
  ...
});

I defined code logic to set nextRetry prop to null when the prop value reaches 10 - such data should not be included in the sync dataset. The query for selecting users for next sync (a simplified version of it):

SELECT *
FROM users
WHERE
    sync_error IS NULL
    OR json_extract(sync_error, '$.nextRetry') < strftime('%s', 'now') * 1000

Explanation: If nextRetry is null, the json_extract function returns NULL. In SQL, any comparison involving NULL (like < or >) results in NULL and the row is not included in the dataset for the next sync. Otherwise, if the current time is past the time of the next retry, the row is included in the result dataset and another sync attempt will be made. Pretty neat, isn’t it?

SQL JSON Extension is very powerful and provides many tools not just for filtering, but also for querying and setting data.

In my case, to update the JSON data I first query the data, then form a new json object in my code and then set it in my table:

UPDATE users
SET sync_error = '{"msg": "...", ...}'
WHERE ...

If I would need to update just one property inside that json object, I could do it directly using json_insert function:

UPDATE users
SET sync_error = json_insert(sync_error, '$.msg', 'error msg')
WHERE ...

In certain situations it might not be performant to use json functions in WHERE or SELECT clauses. To mitigate that performance issue one might create a computed (generated) column to store the extracted value. Computed columns can also be indexed to further improve query performance. It is not needed in my case, so I keep it simple.

Hope my findings can be useful to some.

If there are SQL veterans out there who notice issues with my approach and/or have suggestions, I’m keen to know and learn from you - please provide your opinion and share your experience working with JSON functions.

Prevent ChatGPT from Using Your Data: A Privacy Guide

Alexey Antipov — Tue, 16 Jan 2024 15:04:00 +0000

Last week OpenAI released a new ChatGPT plan - Teams plan. The new plan has one specific feature:

No training on your data

It struck me because I had previously assumed that Plus users’ data was not used for training by Open AI. Apparently, I was wrong.

I use ChatGPT heavily not just for work, but also for personal matters. I don’t feel comfortable knowing that it uses my data for some training and, potentially, shares it with the world.

The data I share with ChatGPT should be private.

I don’t want to use the more expensive Teams plan just for its privacy feature. I wondered if there is any other way of opting out of training?

I dug in and found two ways.

The first approach is using a setting in ChatGPT Data Controls. This is the solution that OpenAI promotes and recommends:

go to “Settings & Beta” ⇒ “Data Controls” ⇒ “Chat History & training” and turn the setting off.

This approach has one significant drawback - it’s coupled with the “Chat History” feature meaning that you can’t turn the training off and keep Chat History. You either have both or don’t. Even more so, this setting is not synced across devices: if you turn it off in your Browser, it won’t apply to you mobile app. More about that setting here.

Everything suggests that OpenAI intentionally designed this opt-out feature in a way that users won't use it, and even if they do use it, then it will work only partially.

The second approach, the real 💪 approach, is not easy to find. It's mentioned only briefly in OpenAI FAQ. What you need is

• go to privacy.openai.com/policies page and submit a “Privacy Request” there.

After submission you’ll soon be notified that your data is no longer used for training. This is exactly what we need: the ability to opt of Training across all devices, while still retaining all of ChatGPT's features.

A year as Staff Frontend Engineer

Alexey Antipov — Fri, 03 Mar 2023 13:00:00 +0000

A year has passed since I joined Flink as a Staff Frontend Engineer.

Now is time to look back and reflect.

Intro

When I was joining Flink, a grocery delivery startup with a multi-billion dollar valuation, I had no idea what the role is about. I bought a book that offered some insights about the responsibilities of Staff engineers, but the scope seemed too vague, I wasn't sure what I should actually be doing in practice and whether that position was right for me.

The change was intimidating and intriguing at the same time.

Now, having worked for a year at that position, I can certainly say that it was one of the best decisions I ever made, and it was probably the most fruitful year of my career.

Over the course of the year, I constantly dragged myself into new areas of work, and I learned and achieved a lot. I built confidence in areas of work that I previously thought were not for me and not in my comfort zone. I discovered that I enjoy and am very good at a lot of other things besides coding. My appetite increased significantly as I made progress.

Learnings

Build relationships and earn credibility

Right from the beginning, I focused on developing trusting relationships with my colleagues, especially with the frontend engineers.

I believed I had to first demonstrate my competence, my ability to write quality code, my ability to provide thoughtful PR reviews, my ability to help find bugs and fix them, and my being a person others can rely upon.

Regular one-on-ones with developers also helped a lot in building relationships. It gave me the opportunity to learn about their struggles, their perspectives on things, and get their feedback on how I performed and how I could help them.

That helped me a lot going forward in implementing my initiatives and getting buy-in from developers.

Don't be a bottleneck

With a good, strong reputation, it's easy to become a bottleneck when people start approaching you directly looking for help. That way, other developers miss an opportunity to help and learn, and you end up spending more time answering questions of all kinds.

Knowledge sharing is important, and whenever someone needs help, other colleagues should have the chance to provide help and learn from one another's mistakes.

I created a dedicated Slack channel for frontend engineers, actively promoted it, and redirected questions there.

Knowledge sharing and building a community

Often, organizational structure doesn't facilitate cross-team or cross-department collaboration. Developers work on their projects in isolation, which negatively affects their company because the same solutions are constantly being re-implemented and the same issues occur across different projects. Lack of collaboration also means fewer opportunities to learn and grow, which leads to dissatisfaction and a poor retention rate.

I had been there myself, and I believed that it's very important to build a strong frontend community where developers could be passionate about their work, share knowledge, support one another, and collaborate on shared libraries and projects.

I organized regular frontend meetups and encouraged developers to present their topics there - to share what issues they have fixed and things they are excited about. I myself regularly shared important information about the company's web landscape, hosting solutions, caching issues, etc.

I was also looking for opportunities to standardize infrastructure in use and create libraries and tools used by all projects across the company. Besides the obvious benefits of infrastructure reuse and code deduplication, this would give developers common ground to work on, provide an opportunity to create an impact on the whole organization, and create a space for collaboration with other developers. Additionally, switching between projects and onboarding new developers would be simplified.

Going together to a conference was another thing I was pushing for. You learn new things, get inspired, and become friends with your colleagues. That, in my opinion, is the best use of the development budget. We attended the React Day Berlin and VueJS Amsterdam conferences.

Focus on impact and your expertize

When you are new to the company, it can be challenging to understand where to start and what to work on. Even company veterans might face the same difficulty.

It happened to me as well. I once joined some backend-focused initiative purely out of interest because I did not see interesting and impactful enough areas in frontend engineering. It was a trap, and I am glad that I escaped it.

What helped me during such moments of doubt was reminding myself that I am a very knowledgeable and experienced frontend engineer and that it would be a shame for the company not to make use of it. There must be something in my area of expertise and interests for me. With that in mind, I explored what was happening in other domains (departments) and what struggles and issues they had. The result of it was me moving to a department in need. I also challenged the status quo in my former department and came up with organizational changes - formation of a web platform team and the transfer of certain projects under its ownership.

Your manager is your advocate

The work you do as a Staff Engineer might not be directly visible to the rest of the company. There can be no direct relationship between your work and the company's performance.

For example, how can you quantify the impact you are making by building a frontend community, setting up a proper interview process, and hiring best-in-class developers?

It can also be challenging for your manager to properly assess your work if they are in charge of a specific domain and your work far exceeds the scope of that domain. They won't be a good advocate for you, and it can hinder your growth and career.

The lesson I learned here is to be conscious of who my manager is and make sure they are interested in the work I do and that my work is transparent to them.

What I like about my work

Freedom

The position gives me freedom - I am in charge of deciding what to work on. I am no longer required to perform tedious tasks, like building ui forms. Instead, I focus on challenging, impactful, and full of learnings tasks that let me fully utilize my knowledge and experience.

Nobody expects me to write code myself and deliver something every day. That gives me time and peace of mind for deep thinking and evaluation of different ideas, to be creative, to work more with people, and to invest my time in them.

A dream job, isn't it?

Power

I am a member of my domain leadership group and my voice is respected. I can influence teams' roadmaps, I can influence the trajectory of web development at the domain level and even at the company level, and I can come up with initiatives for teams to work on.

Freedom and Power combined make me extremely happy and enable me to be as impactful as I can.

Things I wish I had more time for

Writing code

All pre-Staff positions add value to a company primarily by writing code and implementing features themselves. Staff Engineer position bestow Freedom and Power on you to bring far more value. How? By implementing things that scale, i.e., enabling engineers to do their job, helping them, couching them, redirecting their focus to more important tasks, coming up with architectural changes, and bringing highly skilled developers aboard.

As a result, I write less code myself, which is something I wish I could do more of. Although I should be an expert in my field, I don't have as much time for practice now. Finding the right balance is important.

Conclusion

Changing my position from Senior to Staff Frontend engineer was a tremendous shift for me.

It has been a very fruitful, exciting, and full of learnings experience.

I was able to apply my knowledge and experience on a large scale, and my skill set expanded.

Widget Driven Development

Alexey Antipov — Wed, 15 Dec 2021 10:02:48 +0000

Preface

When developing UI applications, we compose them from components. Each UI component is essentially a combination of markup, scoped styles, and some UI logic. Data Management is frequently left out of components control, resulting in a complicated architecture with convoluted data flows.

In this article, I'll demonstrate how we can transform components into autonomous isolated widgets with complete control over both Data Logic and UI.

The History of Components

Widgets, in my opinion, are the natural successors of Components. To see this, I suggest going back in time and look at how our approaches to building UI have evolved over time.

Many remember the times when all application styles were defined in a single global CSS file. Styles definitions used complex combinations of different CSS selectors. Style collisions were common in apps. The size and complexity of those styles sometimes affected even the performance of websites.

In 2009 BEM was born. BEM provided a set of guidelines for defining styles and naming classes. These rules were intended to address the problem of style collisions and inefficient selectors. BEM encouraged to think of UI in terms of blocks, elements, and modifiers.

2013-2015 marked the rise of Components approach. React made it simple to divide UI into components that were a combination of markup (HTML) and UI logic (JavaScript). It was a game changer in application development. Other frameworks soon followed suit, adopting a components-based approach as well.

With the rise of build tools, CSS preprocessors, and techniques like CSS-in-JS and CSS Modules, it became feasible to make Styling a part of Components.

Components playgrounds such as Storybook emerged to help developers build components in an isolated environment and ensure proper styles scoping. They encouraged developers to think of UI as a function of state: components props values defined the look and behaviour of components.

Collections of reusable high-quality components became a thing.

The unresolved hurdles

Component-driven approach helped break UI into isolated reusable pieces and enabled building large-scale applications using collections of pre-built components.

What was missing, though, is a way of supplying UI components with data.

Data management became one of the most difficult tasks in Frontend Engineering and the primary contributor to the complexity of UI apps.

We learned to split components into two types:

Presentational components, which are responsible for UI representation and are usually stateless and side-effect free
Container components, which deal with data-related logic and pass data down to Presentational components.

All that remains is to define how Container components should work with data.

The Naive approach

The naive approach would be for each Container component to simply fetch data needed by underlying Presentational components.

Because the same data is usually needed by multiple different components, implementing such an approach in practice would bring a bunch of problems:

duplicated requests and data over-fetching. Slow UI and overloaded server as a result.
possible data inconsistency between components when requests to the same endpoint result in different data
complicated data invalidation (think of a case when the data has changed on Backend and you need to make sure every dependent component refetches the data)

The Common parent approach

We learnt to overcome the problem by moving data-fetching (and mutation) functionality up to common parent components which pass data down to all underlying components.

We solved the problem of requests duplication and data invalidation. However, we did face new challenges:

the whole application logic became more complex and more coupled
we were forced to pass data down through multiple components. This issue became notorious and got a name “Prop Drilling”.

The State Management approach

To bypass the Prop Drilling problem, we learnt to use State Management libraries and techniques: instead of propagating data down to underlying components, we place data in some Store that is accessible to all the components down the tree, letting them obtain the data directly from there. Components subscribe to the changes in the Store to have the data always up-to-date.

The Prop Drilling issue was resolved, but not for free:

we now have to deal with a completely new concept, the Store, and care about a bunch of new things, such as designing and maintaining Store structure, appropriately updating data in the Store, data normalization, mutable vs immutable, a single store vs multiple stores, and so on.
state management libraries require us to learn a new vocabulary: Actions, Action Creators, Reducers, Middlewares, Thunks, and so on.
the introduced complexity and lack of clarity forced developers to create styleguides on how to work with the Store, what to do and what to avoid.
as a result, our applications became very tangled and coupled. Frustrated developers try to mitigate the issues by inventing new state management libraries with different syntax.

The Naive approach reimagined

Can we do better? Is there an easier way to approach data management? Can we have the data flows transparent and easy to understand? Can we untangle our apps and boost orthogonality? Can we bring Data Logic under control of Components in the same way that we have done with Markup, Styles and UI Logic?

We must have gotten too far into the woods and can't see the forest for the trees. Let's go back to the starting point, to the Naive approach, and see if we can solve its problems differently.

The main bummers there were requests duplication and data inconsistency.

What if we could have an intermediate player between our components and Backend, say an API wrapper or interceptor, solving all those problems under the hood:

deduplicate all the requests
ensure data consistency: all the components should always have the same data when using the same request
provide data invalidation ability: if a component changes data on the server, other components that rely on that data should receive the new data
be transparent to components and not affect their logic in any way (make components think they communicate to Backend directly)

The good news is that we can have it, and there are already libraries providing such solutions:

some GraphQL clients, e.g. Relay
React-Query, SWR, Redux Toolkit Query, Vue Query for RESTful APIs

All we basically need to do is to wrap every API call with such an API Wrapper. The rest is handled automatically for us.

The huge benefit of such an approach is that we can finally untangle our applications' data logic, put Data Logic under control of Components, and achieve better orthogonality by combining all pieces together.

Widget driven development

In my team, we started to use the described above Naive approach together with React Query and we love it. It enabled us to approach building our application differently. Let me call it "Widget Driven Development".

The idea is that we split every page into so-called widgets, which behave autonomously and are self-contained.

Every widget is responsible for:

fetching and providing all the required data to its UI
mutating the related data on server if needed
data representation in the UI
UI for Loading state
(optional) UI for Error state

Speaking of code organization, we co-locate all the widgets related files:

Usually, the same API endpoint is used across multiple widgets. So we decided to keep all of them in a separate shared folder.

We use React Query library and each file in the queries/ folder exposes fetch and mutation methods wrapped into React Query.

All Container components have a similar code structure.

import { useParams } from 'react-router-dom';
import { useBookQuery } from 'queries/useBookQuery';
import { useAuthorQuery } from 'queries/useAuthorQuery';
import Presentation from './Presentation';
import Loading from './Loading';
import Error from './Error';

export default BookDetailsContainer() {
  const { bookId } = useParams();
  const { data: book, isError: isBookError } = useBookQuery(bookId);
  const { data: author, isError: isAuthorError } = useAuthorQuery(book?.author);

  if (book && author) {
    return <Presentation book={book} author={author} />
  }

  if (isBookError || isAuthorError) {
    return <Error />
  }

  return <Loading />
}

Notice how easy and declaratively dependant queries are handled. Also the only dependency of our widget is the presence of bookId in the URL.

Most of our widgets’ container components have no props and rely on no external state except for URL data.

Such an approach makes it transparent what API queries our widget relies upon. That transparency combined with near to zero external dependencies makes it easy to test widgets and gives us confidence in our code.

Usually, changes to a widget are limited by modifications to files under that widget's folder. It significantly limits the risk of breaking any other parts of the application.

Adding new widgets is also very straightforward: create a new folder for the widget with all required files in it and, if necessary, create a new query in the /queries folder. Again, the risk of breaking any other parts of the application is very limited.

Every widget can also be easily reused on different pages thanks to the limited dependency on the context. We usually just need to make sure the URLs of those pages contain the data identifiers needed for the widget.

Conclusion

The Components approach made easy and straightforward creation of reusable independent pieces of UI.
It didn't solve all the problems though and Frontend applications often suffer from convoluted Data Management.

There are libraries that enable approaching Data Management differently and significantly reduce the complexity of our applications.

Leveraging those libraries, we can put data logic under control of components and convert an application into a set of reusable self-contained widgets. It makes the data flows transparent, architecture flexible, the code resilient and easy to test.

State Management: Separation of Concerns

Alexey Antipov — Thu, 11 Nov 2021 13:32:31 +0000

State Management in Frontend is complicated and approaches are not yet settled. New state management libraries keep popping up. In search for a silver bullet, libraries and frameworks authors come up with different brilliant APIs and approaches. Developer Community has produced guidelines for many scenarios.

Nevertheless, developers continue to struggle. Why is that? What do we miss?

The Complexities of State Management

UI is a function of state. We make a minor modification to application state setLoading(true) and the entire UI changes to show the loading indicator.

Doing State Management right is a key ingredient in achieving great UX.

However, it is not a trivial task to do. We need to care about a lot of things:

define the Store structure and what to put in the Store
how and when to initialize and update the Store
immutable vs mutable
local vs global
how to handle dependent state data
how to represent all possible states of API requests
how to mock Store in tests
etc.

As a result, we usually get

a lot of imperative and boilerplate code
components tightly coupled with the Store
components logic that is scattered over multiple files
complicated tests
complicated refactoring
decreased developer productivity

Separation Of Concerns

Developers have created a plethora of libraries, techniques, and guideline to overcome or at least mitigate the challenges. And for many, it introduces a new issue: how to navigate between different libraries and techniques? When to use which?

I recognize an abstraction (perspective) that can be especially valuable when dealing with the subject. This perspective is often missing in discussions about State Management. I am talking about Separation of Concerns.

In UI we deal with huge amount of data with different characteristics and of different nature. We often treat all the data the same way and use the same approaches and libraries.

If we apply the principle of Separating Concerns to state handling, then we discover, that

not all data are equal
some data are simple and trivial to handle
some data are more complex, nuanced and tricky to handle
there are often specialised libraries which help dealing with the complex data

Having realised that, we can start segregating data and look for specialised tools to manage complex and nuanced data. Those tools usually automate a lot of things we used to do manually and bring relief.

I find helpful recognizing the following categories of data:

Server State
Form State
UI State (excl. Form State)

UI State vs Server State

The first and foremost data separation should be made between UI State and Server State data because their characteristics differ greatly.

UI State includes all the data that is not persistent and is not stored on Server.

It is stored locally in Browser and is normally reset on page reload.

The data is synchronous (mostly). Changes to data are “immediate” and there is no intermediate state, we don’t need to wait for new data to come. Whenever the state change happens, we always know the new state.

Most UI State changes are triggered by user actions - “click”, “hover”, “scroll”, etc.

Examples of UI State:

dark/light mode
filters state
forms validation state

Server State data is stored permanently on Server. UI needs to fetch data from Server and send changes back to Server.

Server State is Asynchronous. UI needs to fetch it and that takes time. We don’t know upfront how long it takes and what the value will be. We don’t even know if the request will be successful. The same applies when we need to update the state and persist it on Server.

Another major characteristic of Server Data State - it is remote and it is not under our control. It has shared ownership. Anyone and anything can change data on the Server without our knowledge. It means, that we don’t know for sure if the fetched data is up to date.

Solutions for UI State

Most of existing state management libraries (e.g. Redux, MobX, Zustand) and approaches are tailored to handle synchronous UI State data:

define and initialize the Store
update data in the Store
subscribe to changes in the Store
notify all the subscribed components about the State changes.

Developers just need to choose a proper one.

In my experience, the share of UI state and the code needed to manage it is very small for most applications.

That makes the problem of choosing a UI state management library insignificant. If we use those libraries to handle true UI State data only, then most of them work just fine and the switch between any of them does not make a big difference.

Solutions for Server State

The characteristics of Server State defined above make the handling of it especially difficult and tricky. And that is where common state management libraries don’t help much.

Some of the challenges that developers face when working with Server State:

know if data have already been fetched and is available
know if fetching is currently in progress
know if fetching has failed
deduplicate requests
re-fetch on error
cache data and invalidate the cache
handle mutations with dependent data (think of when changing one entity affects other entities)
optimistic updates
reflect Server State in UI

We all know the cost of attempting to solve these challenges on our own using common state management libraries.

Fortunately, we are seeing a rise of libraries that specialize in managing Server State and solving all the inherent challenges.

https://moiva.io/?npm=@apollo/client+react-query+relay-runtime+swr.

These libraries automate the majority of the tasks, drastically reduce the amount of boilerplate code, and provide declarative APIs with thoughtful defaults.

Some of GraphQL Clients were the first who pioneered the approach. They are designed specifically for GraphQL APIs. Example: Apollo Client, Relay.

Next came libraries to help manage Server State with REST APIs:

At this moment, React Query is probably the most sophisticated and popular library to handle RESTful Server State.

React Query is React specific, but its core was fully separated from React and it can be used to build solutions for other frameworks as well. Such solutions have already begun to emerge. For example, Vue Query.

Form State

It is often helpful to separate Form State handling from the rest of the UI state.

Reason - Form handling is tricky and nuanced. You need:

maintain state of a form as a whole: isDirty, isValid, isSubmitting, dirtyFields, etc.
maintain state of each particular field: isDirty, isValid, errors, isDisabled, currentValue, initialValue
reset form’s and fields’ state
trigger validation
etc.

For simple forms and simple use cases we can manage the state ourselves. But for complex cases it is better to reach out for specialised tools.

Examples of form handling libraries:

Some JavaScript frameworks have built-in tools to manage forms state.

Conclusion

Separation of concerns is an important concept in State Management topic.

Different types of data deserve different approaches and specialised tools.

Using specialized tools often bring huge relief. It’s helpful to be aware of these tools.

Moiva August 2021 update

Alexey Antipov — Tue, 10 Aug 2021 13:45:31 +0000

Hello everyone!

I’m Alexey, creator of Moiva.io, The Best Tool to compare Npm Packages.

This is an August report on the progress of Moiva.io.

I was on vacation in July, so I had plenty of time to implement some exciting features! Let me share them with you.

Copy/download/share charts

This is the feature I’m mostly excited about - all charts now include a popup menu that allows users to copy charts to the clipboard, share charts on Twitter, and download charts.

It should make it easier for users to use the findings of comparisons elsewhere.

A chart for every metric in the Table

A table view is useful for presenting static and numerical data. When the number of libraries in comparison is small, it serves the purpose very well. As the number of libraries increases, it becomes more difficult to make sense of the raw figures. This is where graphical data representation would be very beneficial.

Thinking that way, I added a small icon to every numeric metric in the Table. Clicking on the icon invokes a pop-up window with a bar chart where all the libraries are sorted by the metric’s value.

New Suggestions mechanism

When a user evaluates a library, Moiva suggests alternatives to compare.

In the past, the mechanism of suggestions used Moiva’s Catalog which categorised its libraries. The process of adding new libraries was tiresome because the mechanism had several downsides:

every library had to belong to only one particular category. Choosing a category or creating a new one was a non-trivial task. For example, should React and Vue date-picker libraries belong to one category or different categories? Should Express and NextJS belong to the same category?
the catalog was maintained in a separate repository, each Category had a separate file, and from time to time I had to compile it and put it to Moiva’s repo.

To facilitate adding new libraries, I refactored the Catalog:

to use Tags instead of Categories: each library is assigned a number of tags, not a Category
store the libraries as a plain list in a single file under the Moiva’s repository
use tags of selected libraries to suggest alternatives
sort suggestions based on “similarity” to the selected ones. The more tags a library has in common with the tags of selected libraries, the more “similar” it is.

In the future, I plan to include GitHub Stars and Npm Downloads numbers, and libraries statuses to improve the sorting algorithm.

Tooltips in Suggestions

Before adding suggested alternatives to comparison, it might be helpful to glance quickly at the descriptions and other details of the libraries, such as the number of Stars and Npm downloads.

On that basis, I added tooltips to the list of suggested alternatives. The tooltips contain the following information:

name
description
stars and npm downloads numbers
tags

“About” page

Every respectable web app must have an “About” page. And so does Moiva - https://moiva.io/about/.

Have any feedback or ideas about the project? I’m keen to know!

If you find the project interesting, please consider sharing it.

Vercel Serverless Functions vs Cloudflare Workers

Alexey Antipov — Thu, 25 Mar 2021 07:12:10 +0000

Liquid syntax error: 'raw' tag was never closed

Moiva.io v3: a universal tool to Evaluate, Discover and Compare software

Alexey Antipov — Fri, 19 Feb 2021 11:58:25 +0000

Hi, Alexey is here. I have some exciting news for you!

I rewrote Moiva.io from scratch and made it a Universal and Flexible tool to suit the taste of every software developer be they a JavaScript, Python or [put your favorite language here] developer.

This article marks a third major release of Moiva.

What's new (in short)

ability to search for and get data for any GitHub repository in addition to search and comparison of NPM packages.
possibility to bring (relatively easy) Search, Suggestion, and Comparison capabilities to other programming languages' package management systems like Maven (Java), PIP (Python), or Packagist (PHP).
last but not least, Moiva got open-sourced.

Why I did it

At first, I wanted to focus on JavaScript ecosystem, making npm packages first-class citizens in Moiva.io.

The goal was to provide developers with a good tool to evaluate and compare npm packages in different dimensions - Popularity, Maintenance, Security, etc.

But very soon I realized that there are many JavaScript-related projects which don't have any published npm packages.

Think of, for example, frameworks like Meteor.

Moiva.io could potentially be useful for the evaluation of those projects as well thanks to GitHub charts (Contributors, Issues, Commits Frequency, etc.), but search functionality was limited to npm packages only and everything was built around the concept of npm packages.

On the other hand, if Moiva gets opened up to the search, evaluation and comparison of any GitHub project, it will essentially convert Moiva into a universal tool and make it useful to many more developers.

So I got convinced that Moiva should become more Universal and Agile, I just need to come up with a good harmonious concept of how it should look, work and how to implement it.

AHA moment

In the beginning, the idea of supporting GitHub looked vague and blurred. I didn't have any good idea how to put together existing functionality for npm packages and the new one for GitHub repositories.

I could implement separate pages for npm and GitHub, but that was not ideal. Both have a lot in common when comparing JavaScript projects.

Then the AHA moment came - everything became clear, I realized how to put together different things and since then I was unstoppable.

Here is the essence of the solution.

One Search for All

The same single search field can be used to search for both npm packages and GitHub repositories. It can be easily achieved via search modifiers (prefixes).

The default search is for GitHub.

The search prefixed with n: is for npm packages.

What I like about that solution is that it can be easily extended in the future to search for other things as well.

Show only relevant charts

If a user selects only GitHub repositories without related npm packages, then we can just hide npm-related charts. No reason to show them.

It's similar to how ThoughtWorks TechRadar and Developer Usage charts work - they are shown only when there is data for the selected npm packages.

At the same time, if the user selects a mix of npm and Github projects, we will show npm-related charts for the selected npm packages.

How about URLs

Every comparison a user makes in Moiva should be easily reproducible via URL.

It means that Moiva should be able to derive from the URL what information to load, what to put into comparison.

When npm packages were the only citizens in the Moiva world, the task was solved easily - the selected npm packages' names were listed in a query parameter: https://moiva.io/?compare=react+svelte+vue.

Having 2 types of citizens, npm and Github, where one depends on the other, complicates things a bit. Moreover, we want to build a future-proof solution that can incorporate other types of citizens like PIP and Maven.

GitHub has a broader scope than npm and my first idea was to replace URL npm identifiers with GitHub identifiers. But there are 2 problems with it:

it's not clear how to derive the npm package from the GitHub repository. At least I couldn't find the solution for that.
one GitHub repo can be a source of multiple npm packages. There is no 1:1 connection.

It lead me to the conclusion that GitHub and npm should be referenced separately in the URL.

So I just decided to have separate query parameters: https://moiva.io/?npm=svelte+vue&github=meteor/meteor.

GitHub and NPM reconciliation

Imagine two situations:

a user selects Vue as an npm package.
a user selects Vue as a GitHub repo.

In the first situation Moiva shows npm-related data and charts like npm Downloads. In the second situation, it doesn't.

But is it fair? Most probably a user would expect to see the same set of information in both cases, right?

Could we still somehow derive information about the npm package from the GitHub repository? If yes, then we could show npm data for the selected GitHub repository.

Turns out we can make use of Moiva Catalog which was built to implement the Suggestions mechanism.
For every listed GitHub repository we can add a name of the npm package if there is one. It means we can solve the problem of the reconciliation for items listed in the catalog. And I think it's a good enough solution with which we can cover the most popular libraries.

We just need to take care of some details and edge cases.

If a repository does have an npm package, but that package is just one of the repo's "by-products", then probably it doesn't make sense to show that npm package data when selecting the repository. To solve that problem, an additional flag isNpmCoreArtifact in the catalog can be used to indicate the "role" of the npm package.
If we successfully derive npm data from the GitHub repository, it means we essentially display the same information for both npm and GitHub and have different URL identifiers for the same page. It's not good, especially in terms of SEO. So I decided to use npm package's name as a URL identifier in such cases. Try load https://moiva.io/?github=vuejs/vue URL and see what happens ;=)

Data model

I mentioned just a few of the problems I had to solve. There were, of course, many others, like duplication handling, aliases, SEO, etc.

Most of the problems got a straightforward solution once I implemented a proper Data Model - I came up with a new abstraction called "Library" and provided it with certain properties and behavior.

If you are interested, you can check the repository's readme for more details about the Library concept.

What's next

I clearly see a huge potential for Moiva.io to become a really useful tool to many developers.

It can grow and become better in different directions.
I will mention a few of them which look most important to me:

enable search/suggestion/comparison for more languages' package systems (Maven, PIP, etc.).
add more useful charts and data, both generic and language/package-system specific.
improve significantly the alternatives suggestion system. Currently, it's based on Moiva Catalog and needs a lot of data to be put there. I see a way how the community could help and contribute there.

I hope I didn't waste your time and you found the reading and the project itself interesting.

This article was originally published on Moiva Blog https://moiva.io/blog/universal-tool-to-evaluate-discover-compare-software