DEV Community: A R T I U M

Memory management and other weird blobs

A R T I U M — Wed, 26 Jun 2024 21:39:59 +0000

I always felt confused by the terms “Blob, Buffer, ArrayBuffer, Uint8Array.”

Everyone has their own definition, but no one can clearly define them. It always ends up in a nonsensical technical explanation.

Today, I’ll put a human-readable definition on these terms. This will help my future self and other engineers to have a better understanding.

Let me introduce you to the strange world of memory management in JavaScript.

Before we dive into the subject — if you have any questions during the lecture — feel free to ask me on Twitter: @artiumWs

Concepts

Before starting, let’s define some basic concepts: Memory, Memory Span, Binary Data, and Raw Data.

Memory

Memory (RAM) is fast storage used by a computer to quickly access and store data.

Memory span

A memory span refers to a contiguous block of memory.

If memory is a street, a memory span would be neighbouring houses.

Binary data

Binary data is the representation of information in binary form — bunch of 0 and 1

Raw Data

Raw data is the unprocessed information consumed by a program — text, images, files …

Keep these definitions in mind. They will be useful for understanding the following sections.

In-code object

Blob object

A Blob is an immutable object referencing raw data.

This is an object created in your code that contains raw data information like size in bytes and MIME type (image/png, application/pdf, application/json, …).

File object

A File extends from Blob properties and is created when using an input element.

It’s a Blob with more information like: last modified date and file name. This object is automatically created when a file is uploaded to your app using an input element or a drag-and-drop event.

Those object cannot be modified directly! They are the in-code reference to the raw data.

This is to ensures their integrity when manipulating them.

Manipulation

Those objects provide two methods to access the data: arrayBuffer and stream

arrayBuffer

arrayBuffer is method that load the binary data of the object in a memory span and return a in-code representation of a memory span.

This representation cannot be modified directly! We need to reference it first with a typed array to access its binary data. Typed arrays are defined later.

Caution: It loads the whole file in your memory at once. In case the file size is to big (>5Mb), this operation can freeze your app. In such case, use the next method.

stream

stream is a method that load the binary data in chunks. This process is called buffering.

Chunks returned are directly typed arrays so they can be directly manipulated to process you data on fly.

Typed array

Typed array (uInt8Array, int16array …) references the arrayBuffer to access and modify its binary data.

Multiple typed array can reference the same arrayBuffer. In such cases, the latest modification will overwrite the previous ones.

Bonus

Buffer (Node.Js)

Buffer is a Node.js object that extend from the typed array properties. It is designed to deal with communication features.

Use case:

Buffers are designed to handle communication logic.
eg: Optimise data send through requests.

Typed arrays are designed to handle processing logic.
eg: Adding a filter to an image

Recap

Let’s recap this article through the following use case: “Uploading an image on a web app”

An image is stored on the your computer (hard drive, SSD, …)
Upload this image to your web app using an <input type="file"> element
The browser creates a File object that references the image.

4.a Use the arrayBuffer method to load the image into memory span and reference it
5.a Attach a typed array to this arrayBuffer to read or write binary data
6.a Process the binary data

4.b Use the stream method to read chunk of stored image as a typed array
5.b Process the binary data chunks

I hope this article helps you gain a clearer understanding of the topic.

If you have any questions, feel free to ask me on Twitter: @artiumWs

Sources

https://nodejs.org/api/buffer.html
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Typed_arrays
https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Concepts
https://developer.mozilla.org/en-US/docs/Web/API/Streams_API
https://developer.mozilla.org/en-US/docs/Web/API/File
https://developer.mozilla.org/en-US/docs/Web/API/Blob
https://stackoverflow.com/questions/11821096/what-is-the-difference-between-an-arraybuffer-and-a-blob#comment41489269_11821109
https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream

Backblaze B2: How to store a file

A R T I U M — Sun, 16 Jun 2024 19:15:28 +0000

Backblaze B2 is a cloud storage solution offering scalable and affordable data storage for individuals, developers, and businesses. Backblaze's B2 service aim for simplicity and cost-effectiveness.

However, users may find the documentation cumbersome to understand, which can be a hurdle for those new to the platform.

Intro

This article introduce how to use the backblaze cloud storage solution to store private user file that can only be accessed by the owner.

This article will cover:

How to setup a bucket for your application
How to authorise your application to access the bucket
Upload a file to your bucket
Download a file for a user with the right access

Bucket

When setting up the bucket make sure the following options are checked:

Private This will ensure your files won’t be accessible without the correct permission
Encryption This will ensure your file are properly encrypted. Make sure there is no sensitive informations in the name or metadata of the files. Those won’t be encrypted!
Object lock (optional) This will ensure your file won’t be altered for the selected timeframe. In my case that wasn’t needed so I left it unchecked

Application keys

To access your bucket you will need to setup application keys. You can use the master application keys but I won’t advise it as it have all right for all bucket.
You should consider creating a dedicated application key for your bucket this will prevent any interaction with other buckets.

Once created you should have access to your application key id and application key.

Save them for later use as they will be hidden as soon as you leave the page!

In case that happened no worries just delete and recreate your application key.

Authorise your application

To setup authorisation in your application you will need to fetch the following endpoint.

Example
https://api.backblazeb2.com/b2api/v3/b2_authorize_account

Authorisation header
To access this endpoint you will need the application key id and application key setup earlier. Concatenate them in a string, encode them to Base64 and prefix them with Basic .
Here is a pseudo code example:

Basic Base64({application_key_id}:{application_key})

Returned data
This endpoint will return data related to your application key. Here is the list of required datas for the next step:

Api url
Bucket id
Bucket name
Authorisation token (1) from application key
Download url

More infos
https://www.backblaze.com/apidocs/b2-authorize-account

Upload a file

Uploading a file is done in 2 steps:

Get the upload url
Upload the file

1) Get the upload url

This endpoint will return you the data needed to upload your files. You will need the Api url, Bucket id and Authorisation token from the previous endpoint.

Example
{api_url}/b2api/v3/b2_get_upload_url?bucketId={bucket_id}

Authorisation header
Pass the Authorisation token from application key to the Authorisation header:

Authorisation {authorisation_token_1}

Returned data
This endpoint will return data required to upload a file:

Upload url
Authorisation token (2) to upload a file

More infos
https://www.backblaze.com/apidocs/b2-get-upload-url

2) Upload a file

This endpoint will upload a file to your bucket. You will need the Upload url and Authorisation token from the previous endpoint.

Example
{upload_url}

Use the url previously returned.

Authorisation header
Pass the Authorisation token to upload a file directly to the Authorisation header:

Authorisation {authorisation_token_2}

Headers
You will need to pass header related to your file metadata.

X-Bz-File-Name The name of your file in the bucket. NB: In case you want to separate your file per users you can use this syntax: {user_id}/{file_name} This will store the file related to your user in a dedicated folder to prevent other users accessing it.
Content-Type The MIME type of your file. eg: image/png
Content-Length The size of you file in bytes.
X-Bz-Content-Sha1 The Sha1 of your file to ensure integrity of the file. This need to be generated in your code. In development environment you can pass do_not_verify this will skip the integrity validation.

More infos
https://www.backblaze.com/apidocs/b2-upload-file

Download a file

Downloading a file is done in 2 steps:

Get the download authorisation
Get the file

1) Get the download authorisation

This endpoint will generate a download authorisation for a / some files. You will need the Api url, Authorisation token and Bucket id from the first authorisation endpoint.

Example
{api_url}/b2api/v3/b2_get_download_authorization?bucketId={bucket_id}

Authorisation header
Pass the Authorisation token from application key to the Authorisation header:

Authorisation {authorisation_token_1}

Query params
This endpoint will require some more query params to work properly:

fileNamePrefix The prefix (folder) previously used to store our files. In our case that was the user_id . This will generate authorisation only for the chosen folder.
validDurationInSeconds The number of seconds before your authorisation expire.

After passing those query params your endpoint will look something like:

{api_url}/b2api/v3/b2_get_download_authorization?bucketId={bucket_id}&fileNamePrefix={user_id}&validDurationInSeconds={duration_in_sec}

Returned data
This endpoint will return data required to download a file:

Authorisation token (3) to download a file
File name prefix in our case user id

More infos
https://www.backblaze.com/apidocs/b2-get-download-authorization

2) Get a file by name

This endpoint will download a file from your bucket based on its name. You will need the Download url, Bucket name from first authorisation endpoint. Authorisation token and File name prefix in our case user id from previous endpoint.

Example
{download_url}/file/{bucket_name}/{user_id}/{file_name}?Authorization={authorisation_token_3}

This link can be used to fetch a file for a specific user with an expiration limit.

More infos

https://www.backblaze.com/apidocs/b2-download-file-by-name

Schema

To have a better overview of the implementation here is an explication schema:

Modular Monolith: A disruptive guide to architecting your React app

A R T I U M — Sun, 09 Jun 2024 18:52:19 +0000

Intro

This writing was inspired by the article: “How to Structure Vue Projects”.

It explains various front-end application structures. It’s really insightful and can be applied to other front-end frameworks.

A year ago, my team and I started a new project from scratch. The discussion about its structure and stack became crucial. We decided on a modular monolithic architecture.

I will detail what motivated our choices and hope these insights will inspire you for your future projects.

context

Through my experience, I have worked on several projects. Each with its unique specificities which can be classified into these categories:

Monolithic

A monolithic architecture is an all-in-one project containing all application functionalities.

Advantages: Easy to onboard new team members, maintain, update dependencies, and add new features. Deployment is straightforward and depends on one CI/CD pipeline.
Disadvantages: As the engineering team grows, it leads to more conflict management. Deployment can bottleneck when two teams try to release simultaneously. There is no clear ownership of features.

In short, it doesn’t scale well.

Micro-front end

A micro-front end architecture is a spliced application where each feature lives in a separate project, owned by a dedicated team.

Advantages: Each team can add new features without conflict or deployment bottlenecks. Each micro-front end has its own CI/CD pipeline, allowing specific actions for each project.
Disadvantages: Harder to onboard new team members and update dependencies. It requires more configuration and maintenance, often necessitating a dedicated DevOps team.

In short, it scales well — until it doesn’t.

What happens when you need to scale down?
In 2023, team reductions due to layoffs lead to shared ownership and less-maintained parts. Usually pipelines and configurations, leading to poor developer experience and slowing down the entire process.

I wouldn't advise any company smaller than FAANG size to go for it.

With that said, let’s introduce the architecture we decided to use.

Challenges:

Enabling multiple teams to work on the project without conflicts
Maintaining the project by one or several teams
Having a single project that is easy to maintain
Ensuring low complexity for easy onboarding
Scalable architecture that can scale both up and down

Modular monolith

A modular monolith is an all-in-one project where each feature lives in a separate shell.

In other words, each feature has its own folder and does not share any components, state, or logic with other features. If shared logic, state, or components are needed, they are stored at the root of the project.

This architecture is flexible and can be adapted to your stack and needs.
For example, you can add layouts if you need layouts for your pages or containers if you are working with Redux.

Architecture details

I will detail the choices made for our stack and the conventions implemented to address the challenges.

Conventions

Avoid the trend tech syndrome: New JavaScript libraries are released frequently, making it hard to stay focused on a stack. It might seem like there's always a library that solves an issue in a more elegant way. Don’t fall into this trap. Stick to the stack you have defined.
Avoid nesting in components folder (The atomic design trap): It's common for people to store components related to a parent component within its folder. Leading to a very nested structure that is hard to navigate for newcomers. Prefer a flat structure within your components folder.

Put Logic in Hooks: Start by writing the logic within your component. When the logic takes up too much space, move it to a hook. This approach is subjective and may vary for each individual.
Place testable logic in utils: Separate logic that requires native hooks (useState…) from the logic that doesn't. Move the latter to a utils file if it needs to be tested. More details about this are in the testing section.
Maximise the utility of third-party dependencies: Avoid adding a dependency that can handle multiple things if you only use one of its features. For example, using RxJS to handle simple HTTP requests when the Fetch API can do it just as well.

Stacks

Shared state (`stores` in structure diagram)

Shared state management is always a highly opinionated topic. There are different preferences based on various factors:

Redux: Preferred by those who like a stricter pattern, a strong community, and solid debugging tools.
MobX: Preferred by those who prefer more freedom in implementation and a fine-grained updating system to improve optimisation.
New Trends (Jutai or Zustand): Preferred by those who enjoy new trends because they are hook-like and easy to implement.

Our choice was React Context.

Some may argue that React Context, if not memoized, will re-render all children, and the syntax can be cumbersome.

They would be right.

However, it is native to React, has a low learning curve, and in version 19, the syntax will be simplified.

The Virtual DOM (VDOM) is performant, and re-rendering all children rarely causes issues. If it does, just memoize it properly. React has announced their new React-Compiler (see React-Compiler: When react becomes Svelte), which will optimise during build time.

Styles

CSS-in-JS (styled-components)

Should I really explain why it’s a bad idea ?

Size of the documentation: Why do we have migration guide when dealing with CSS ?

JS handling CSS: Why Javascript would be better than CSS at handling CSS ?
Component instead of class: Why do we need to create a component to apply border styles ?

Don’t recommend to use it in a project

Tailwind

Inline styling done with elegance

Locality of behaviour: Styles are directly within the component
Easy syntax: Simple and intuitive
Optimised: Minification and network compression

I highly recommend Tailwind for any project.

However, newcomers may face a small learning curve to memorise each classes. To onboard people quickly and easily, we decided to go with the following:

Inline-style

The easiest way to handle CSS

Locality of behaviour: Styles are directly within the component
Super easy: Straightforward and quick to implement

If we needed selectors or animations, we used:

CSS modules

Namespaced css

Local to your component: Namespaces classes to your component
Native css syntax: Uses regular CSS syntax

Tests

Unit test (components)

I strongly believe that unit testing components is a waste of time and energy.

Components will be tested manually with or without unit tests. Testing things that can be easily spotted by eye is meaningless.

The only case where I’ve seen a benefit is when working on a UI library that serves several projects.

Unit test (logic)

As previously mentioned, if you need to test logic, extract it from a custom hook and place it in a utils file. Separate the logic from native hooks (e.g., useState, useEffect) and test the logic itself.

Example:
If a custom hook formats data before storing it in a state, extract the format data logic into a utils file and test it there.

Testing a function is easier than testing a hook.

End 2 end testing

I strongly believe that unit testing components is a waste of time and energy because of E2E testing.

Testing a user scenario covers more than just components. It checks if all features of your application are well-implemented together.

E2E testing ensures that critical user paths are working properly: logging in, buying, liking, sharing, etc., depending on your application's business model.

These tests are directly linked to the value of your product. It is more crucial than making sure a FAQ button dispatches an event properly.

conclusion

Each project has its own specificities, and each team has its familiarities with stacks. This architecture and stack won’t fit every project.

The best stack is the one you know

An expert in JavaScript will build a better app in JavaScript than in Rust, even if Rust is known for its performances.

We were a team of five (designer, product manager, engineering manager, and engineers) working on this project. This was aimed to be an internal tool used by our company and it works great.

The workflow is really smooth, and other teams can implement features without difficulties.

The success of choosing a architecture for your project relies on:

What you want to achieve with it ?
Who will work on it ?
Who will maintaining it ?
Are all member of the team aligned with it ?

I hope you enjoyed this article.

If so, don’t hesitate to keep in contact with me:
Stay in touch on X (Twitter)

If not, please feel free to add your critiques in the comments.

Sources

What is locality of behaviour:
https://htmx.org/essays/locality-of-behaviour/

What is modular monolith:
https://www.milanjovanovic.tech/blog/what-is-a-modular-monolith

How to structure a vue project:
https://dev.to/alexanderop/how-to-structure-vue-projects-20i4

Documentation about react-hooks-testing-library:
https://github.com/testing-library/react-hooks-testing-library

Documentation about styled components:
https://styled-components.com/

Documentation about atomic design:
https://atomicdesign.bradfrost.com/chapter-2/

How tailwind optimise for production:
https://tailwindcss.com/docs/optimizing-for-production

What is css modules:
https://github.com/css-modules/css-modules

React-Compiler: When React becomes Svelte

A R T I U M — Sun, 02 Jun 2024 17:34:17 +0000

Introduction

The React ecosystem is evolving, and the React Compiler is a game-changer. Here’s a comprehensive look at how this new tool enhances performance by automating optimisations.
Check out my insights and predictions.

I believe hyperlinks in articles can be distracting. You will find the sources that helped my reflections at the end of this article.

With that said, let’s dive into the context.

React

In 2013, browsers were inefficient at updating the DOM. The React team developed a new approach called the Virtual DOM (VDOM). Elements are first updated in the VDOM before being passed to the actual DOM.

Reactivity in React is handled at run time. When a component re-renders, it updates the view along with its nested components. This process can be optimised using techniques such as hooks (useCallback and useMemo) and memoization (memo).

Svelte

By 2016, browsers had evolved, and updating the DOM directly was no longer as slow. Svelte compiles the code into optimised JavaScript that interacts directly with the DOM.

Reactivity in Svelte is handled at build time. Svelte compiler binds state to elements, allowing for more fine-grained updates.

Reactivity

Everyone seems to have their own definition of it. So here is mine:
Reactivity is the way the view responds to a change in the state

The relation between state and the view is as follows:

A state is logical representation of a view
A view is visual manifestation of a state

Let’s consider the following example:

Here is how both frameworks will handle reactivity if a new task is added to the list:

React
React will update the parent (TodoList) state (todos) to add a new task. It will re-render the parent which will trigger each child (Task) to re-render.

This operation can be optimised with hooks to prevent unnecessary re-renders.

Svelte
Svelte will add an new items to the state and append a new DOM element to the list.

No optimisation required.

React compiler

The previous example highlights the need to optimise React code to prevent unnecessary re-renders. As a developer, it feels like you need to handle the framework's optimisation yourself.

This can be cumbersome and lead to two patterns I’ve observed in my career:

Those who never use optimisation hooks. Waiting for a optimisation issue to appear and investigating and fixing them (my approach)
Those who overuse optimisation hooks. Wrapping all functions and computations into hooks to reduce the chance of optimisation issues. Reducing at the same time the readability and maintainability of the code base.

Those twos opposing approaches often lead to arguments in PRs, slowing down the overall workflow.

React compiler solve this issue by analysing your code and applying optimisation at build time.
An interesting aspect is optimisation will use Javascript primitives and not hooks.

the compiler memoizes using low-level primitives, not via useMemo

If you already used hooks to optimise your code base. Don’t worry, the compiler will keep your optimisations, allowing you to implement it granularly.

One last thing to keep in mind is that the React Compiler will skip components that don’t respect React rules. To ensure your codebase respect these rules, you can add this linting rule — react-compiler/react-compiler — created for this purpose.

This will ensure your codebase is future-proof and can use React Compiler once it’s released.

Future

The first version of React compiler focuses on hooks (useCallback and useMemo) and memoization (memo) to facilitate optimisation. It optimise the code using Javascript primitives to achieve what they call "fine-grained reactivity".
This approach is very similar to what Svelte compiler already provides.

In the future, React may not only optimise during compilation but also build an optimised Javascript that interacts directly with the DOM, removing the VDOM abstraction leading to improved performances.

I hope you enjoyed this article.

If so, don’t hesitate to keep in contact with me:
Stay in touch on X (Twitter)

If not, please feel free to add your critiques in the comments.

Sources

As promised, here are all the sources for more information on the topic.

What is the Virtual Dom:
https://www.geeksforgeeks.org/reactjs-virtual-dom/
Conference about React philosophy:
https://www.youtube.com/watch?v=x7cQ3mrcKaY
Why Virtual Dom is not the best approach:
https://svelte.dev/blog/virtual-dom-is-pure-overhead
Conference about Svelte philosophy:
https://www.youtube.com/watch?v=AdNJ3fydeao
React-compiler linting rules:
https://www.npmjs.com/package/eslint-plugin-react-compiler
React-compiler doc:
https://react.dev/learn/react-compiler
React-compiler github discussion:
https://github.com/reactwg/react-compiler/discussions/5

DEV Community: A R T I U M

Memory management and other weird blobs

Concepts

Memory

Memory span

Binary data

Raw Data

In-code object

Blob object

File object

Manipulation

arrayBuffer

stream

Typed array

Bonus

Buffer (Node.Js)

Recap

Sources

Backblaze B2: How to store a file

Intro

Bucket

Application keys

Authorise your application

Upload a file

1) Get the upload url

2) Upload a file

Download a file

1) Get the download authorisation

2) Get a file by name

Schema

Modular Monolith: A disruptive guide to architecting your React app

Intro

context

Monolithic

Micro-front end

Challenges:

Modular monolith

Architecture details

Conventions

Stacks

Shared state (stores in structure diagram)

Styles

Tests

conclusion

Sources

React-Compiler: When React becomes Svelte

Introduction

React

Svelte

Reactivity

React compiler

Future

Sources

Shared state (`stores` in structure diagram)