DEV Community: Gabriel Belgamo

How having a Data Layer simplified Offline Mode in my frontend app - Part 1

Gabriel Belgamo — Mon, 15 Jul 2024 14:02:07 +0000

You're done with a project and the product team comes to you and says: "Hey, I turned off my internet and the app died". You stop for a moment and don't recall seeing this requirement throughout the development. You're afraid because whatever you answer will make them frustrated. Well, there's clearly a lack of communication but you're as guilty as them for not having raised that up before.

The Challenge

Our project was a very traditional web app fully dependent on a network connection to work. The app wasn't designed to work offline at all and we hadn't followed any of the offline-first patterns. When a route was accessed, a request was dispatched to get the data needed for that screen. If it succeeded, its result was stored on the local state of the component that was then rerendered. Pretty simple, right? But it was also going against the offline-first good practices. There was a good thing though, the architectural decisions we made. We'll be talking about it in a bit, but let's focus on the problem first of all.

Partial vs Full Offline experience

We faced a clear conflict with the product team that needed resolution ASAP. To product people, enabling offline mode might seem as simple as pressing a button, expecting everything to work out of the box. However, they often overlook that a lack of internet inherently limits the app's functionality. To address this, we scheduled a meeting to clarify their offline requirements. Did they need the entire app to work offline or just certain parts? Should all features be available, or only the essential ones?

This is crucial and my first advice to you: thoroughly understand what the stakeholders want so you can make the right decisions. Failing to get this right initially can lead to back-and-forth discussions. With clear answers, you can determine the right approach for enabling offline capabilities.

In our case, we discovered they only needed offline functionality for a specific part of the app used by one user group. This realization simplified our task by at least 30%. Additionally, most writing operations could be disabled offline, allowing us to avoid the complex process of reconciliation/synchronization. Ultimately, it's about managing trade-offs and finding the best middle-ground solution.

The Data Layer to the Rescue

One of the most powerful patterns when it comes to front-end development is to try to keep your UI as dumb as possible. There are several benefits in doing that such as easy testability, debugging, and the possibility to inject dependencies into your components. This last one in particular was what saved our lives because the data were abstracted into a completely separate layer and we were connecting it with the UI layer through dependency injection. It looked something like this:

So in other words, there was an interface which was the only thing the UI layer knew about and the RemoteDAO was being injected into it. Well, that being said we could simply create another implementation of the DAO replacing the RemoteDAO. However, this new implementation would communicate with local sources of data instead of trying to hit the remote endpoints. We would also need a caching step which is when we store the remote data locally, something like this:

The implementation

To show you a practical example, I built a small todo application using Lit + Vite, but it applies to any framework. Think of an app where parents can assign tasks to their children but some tasks need to be done out of home so they might face poor connection or not have any connection at all. That's why we need to leverage offline features.

On the app entry point, I inject my RemoteTodosDAO instance by using Lit context (similar to React context or a DI container):

@customElement("main-app")
export class App extends LitElement {
  @provide({ context: TodosDAOContext })
  remoteTodosDAO = new RemoteTodosDAO();

  ...
}

Then, I'm able to get this instance from any component by doing:

@customElement("todos-page")
export class TodosPage extends LitElement {
  @consume({ context: TodosDAOContext })
  todosDAO!: TodosDAO;
}

By the way, here's is what the RemoteTodosDAO looked like:

import axios from "axios";
import { Todo, TodosDAO } from "./types";

export class RemoteTodosDAO implements TodosDAO {
  private _httpClient = axios.create({ baseURL: "http://localhost:3000" });

  async list() {
    return (await this._httpClient.get<Todo[]>("/todos")).data;
  }

  async get(id: string) {
    return (await this._httpClient.get<Todo>(`/todos/${id}`)).data;
  }

  async save(todo: Todo) {
    return (
      await this._httpClient.patch<Todo>(`/todos/${todo.id}`, {
        ...todo,
      })
    ).data;
  }
}

Caching data locally

I've mentioned our app was triggering requests as necessary when the user got to some route. But since we must cache all data on the first load, we decided to fetch all needed resources (hitting all endpoints) right when the user accessed the app for the first time. By doing that, we don't have to change anything in the UI layer and we can continue triggering requests when the component is mounted. But instead of reaching the remote source, the app will get the cached data.

As an alternative to hitting all endpoints at once, you could also create a single endpoint on the backend to get the data in a one-way trip. But remember to take performance into account. Are your users mostly on 4G connections or Wifi? How heavy is your data? You'll want to return as little data as possible to not waste resources, return only the most recent ones if your scenario allows so, or even implement pagination.

Now we're going to create a new class to interact with the IndexedDB using Dexie, a wrapper that makes it easier to create schemas. It works the same way as the DAO, but I'm going to name it "Persistor" for convenience. I'm also including the word "progress" in the table name on purpose, I'll explain why in the second part of the article.

const TodosProgressLocalDB = new Dexie("TodosProgress") as Dexie & {
  todos: EntityTable<Todo>;
};
TodosProgressLocalDB.version(1).stores({
  todos: "&id",
});

export class DexiePersistor implements TodosPersistor {
  async saveBatch(todos: Todo[]) {
    await TodosProgressLocalDB.todos.bulkPut(todos);
  }
}

Next, let's update our RemoteTodosDAO to receive an instance of the persistor. Given that this app is fairly simple and the GET /todos endpoint returns all the necessary data, we’ll pass its response to the persistor’s saveBatch method. However, if your app requires multiple requests interacting with different persistors, you can adjust accordingly.

Essentially, every time the user accesses the app, we will invalidate the local cache by rehydrating it with fresh data. To optimize performance and reduce backend load, consider some invalidation strategies to avoid hitting the backend on every single request.

export class RemoteTodosDAO implements TodosDAO {
  private _todosPersistor: TodosPersistor;

  constructor(todosPersistor: TodosPersistor) {
    this._todosPersistor = todosPersistor;
  }

  private _httpClient = axios.create({ baseURL: "http://localhost:3000" });

  async list() {
    const data = (await this._httpClient.get<Todo[]>("/todos")).data;

    this._todosPersistor.saveBatch(data);

    return data;
  }

  async get(id: string) { ... }

  async save(todo: Todo) { ... }
}

As long as they land in a route within the app, the request will be triggered and the response cached. One of the drawbacks of this approach is that it increases the first-load time. So make sure you show some kind of spinner to your user to indicate the page is loading. Something like this:

@customElement("main-app")
export class App extends LitElement {
  todosPersistor: TodosPersistor = new DexiePersistor();

  @provide({ context: TodosDAOContext })
  remoteTodosDAO = new RemoteTodosDAO(this.todosPersistor);

  private outletRef = createRef();

  @state()
  caching = true;

  async firstUpdated() {
    try {
      await this.remoteTodosDAO.list();
    } catch (error) {
    } finally {
      this.caching = false;
    }

    makeRouter(this.outletRef.value as HTMLElement);
  }

  render() {
    return html`
      ${this.caching ? html`<p>Caching resources...</p>` : null}
      <div ${ref(this.outletRef)}></div>
    `;
  }
}

If you check the application tab through the browser inspector, you're going to notice that some data has already been recorded into IndexedDB collections which means we're ready to go to the next and last step to make the reading part work.

Replacing the DAO implementation

All we need to do now is create the new implementation of the TodosDAO interface, which is going to be used as a replacement for the current one.

export class LocalTodosDAO implements TodosDAO {
  async list(): Promise<Todo[]> {
    return await TodosProgressLocalDB.todos.toArray();
  }

  async get(id: string) {
    return await TodosProgressLocalDB.todos.where("id").equals(id).first();
  }

  save(todo: Todo): Promise<Todo | undefined> {
    throw new Error("Method not implemented.");
  }
}

I left the save method not implemented on purpose as well because I'll be clarifying it in the second part of this article. Alright, let's replace the implementation:

export class App extends LitElement {
  todosPersistor: TodosPersistor = new DexiePersistor();

  remoteTodosDAO = new RemoteTodosDAO(this.todosPersistor);

  @provide({ context: TodosDAOContext })
  localTodosDAO = new LocalTodosDAO();

  ...
}

Now you should be able to turn off your backend server and the app will keep working normally, except for the writing operations.

PWA & Routing concerns

Regarding caching, there are basically two types of resources: assets and data. We just handled data but in order for your app to really work offline you need to cache the static assets such as HTML, CSS, and JS by using a service worker and a manifest file. That's a simple step and if you're using Vite you can achieve that pretty easily with Vite PWA.

You might need to tweak your routes as well if you're using code-splitting. Code-splitting essentially fetches the assets on-demand for a given route to work. However, you don't want your user accessing all the routes in your system beforehand just to get them cached, right? So make sure you disable that, at least for the routes you want to work offline.

Conclusion

Offline mode isn't a trivial topic. It's hard enough when developing a PWA from scratch following offline-first good practices; adding it to an existing app is even harder. However, it's possible to work around and achieve good results if you keep things uncoupled and fully understand the real customer needs. As demonstrated, we managed to get the app functioning offline without altering the UI layer. In the second part, we'll explore how to enable users to perform writing operations offline.

The example code is available on GitHub https://github.com/belgamo/partial-offline-poc

How to improve filtering on a product listing page with vanilla JS and React

Gabriel Belgamo — Thu, 01 Jun 2023 10:23:10 +0000

A few days ago I was asked by my teammate designer to “tweak” our sidebar where the filters were placed. It looked pretty simple, however I realized how hard it was when I started coding and figured out that only CSS was not enough. Of course, as developers we usually get angry when designers ask us to create stuff that from their perspective is a no-brainer, but I had no words to argue when I saw his idea in action. That was a pretty fancy sidebar and I got surprised because I haven’t found popular e-commerces taking this approach.

Most of e-commerces have a grid of products and a sidebar with options so the users can refine their search. But what if you have a hundred products, will your sidebar go along with the grid as you scroll down? You might be wondering “alright, just define the position as sticky and that’s it”. You’re not wrong, but what if you have several filter options that don’t fit the screen height? If we place the sidebar at the top using position: sticky the users wouldn’t see all filters that overflow the viewport:

Also adding a scroll auto; to it is not a good solution because this would result in two columns with independent scrolls. That’s exactly the problem we were facing.

Solution

In order to fix that, the designer proposed our sidebar to be always present no matter how long we had scrolled and whenever we reached the start or the end of the sidebar, it should get sticky. The goal here is to make the sidebar scroll with the grid so the users can use the filters without having to go all the way up to find them. Cool, ain’t? Before getting into the code, let’s summarize the logic to avoid getting lost:

If the sidebar is shorter than viewport, it should get sticky to the top
If the sidebar is taller than viewport
- It should scroll with the grid
- And scrolling down
  - It should get sticky to the bottom when reaching the end
- And scrolling up
  - It should get sticky to the top when reaching the start

To accomplish this, we’re gonna have to toggle between the sticky and relative CSS positions and then calculate the top distance dynamically. Looks like someone has faced this problem and made a package for that. By the way, that's where I grabbed the post image from. I didn't even try to use this library because it's not being maintained and probably has more code than we need. Considering it's from six years ago and thus doesn't implement the sticky position, I decided to come up with my own implementation by doing it in the react way using refs and a custom hook, but you can adapt the code to your needs. I’ll assume you have a two-column grid being the first one the filters sidebar and the second one the list of products, or something similar to this. That’s our basic structure to make it work:

<main className="products-list">
  <aside className="products-list__sidebar">
    <Filters />
  </aside>
  <section className="products-list__list">
    { // your products }
  </section>
</main>

.products-list {
  display: grid;
  grid-template-columns: 1fr 4fr;
}

Let’s get started by creating a new custom hook to abstract all logic related to this. I’ll name it useStickySidebar. The hook will receive a reference to the target element, in our case, the sidebar. Since we want to toggle the sidebar position, we need a factor to decide when it should happen. We can do that by listening to the scroll event and then whenever the user scrolls through the page, we trigger a handler function:

const useStickySidebar = ({ targetRef }) => {
  const sidebarRef = targetRef;

  const calculateAndApplyPosition = useCallback(() => {
    console.log({ sidebarRef });
  }, [sidebarRef]);

  useEffect(() => {
    window.addEventListener(
      "scroll",
      calculateAndApplyPosition
    );

    return () => {
      window.removeEventListener("scroll", calculateAndApplyPosition);
    };
  }, [calculateAndApplyPosition]);
};

Now create a ref and attach it to your <Filter /> component and instantiate the hook:

const sidebarRef = useRef(null);

useStickySidebar({ targetRef: sidebarRef });

<main className="products-list">
  <aside className="products-list__sidebar">
    <Filters ref={sidebarRef} />
  </aside>
  <section className="products-list__list">
    { // your products }
  </section>
</main>

Caveat

Be careful when listening to the scroll event and avoid updating react states. You might fall into performance issues!

Once it's done, you should be able to see your console logging the node ref. Now we're gonna add three methods to our hook that will be responsible to make our sidebar to become sticky or relative:

const attachSidebarToTheTop = useCallback(() => {
  const sidebar = sidebarRef.current;

  sidebar.style.position = "sticky";
  sidebar.style.top = "0px";
}, [sidebarRef]);

The first one is pretty simple. Basically, it covers one of our requirements which is the standard behavior when the sidebar is smaller than the viewport.

const attachSidebarToTheBottom = useCallback(() => {
  const sidebar = sidebarRef.current;

  const { innerHeight } = window;

  sidebar.style.position = "sticky";
  const top = innerHeight - sidebar.offsetHeight;
  sidebar.style.top = `${top}px`;
}, [sidebarRef]);

The second one does pretty much the same, but the idea here is to make the sidebar to become fixed at the bottom. We do that by calculating the difference between the viewport and the sidebar height. Most likely you will get a negative result, which is what we use to pull the sidebar up so that we see the last filter item. This covers the part where we reach the end of the sidebar but haven't reached the end of the products grid. So the product grid keeps scrolling while the sidebar is fixed at the bottom.

const makeSidebarScroll = useCallback(() => {
  const sidebar = sidebarRef.current;
  const stickySidebarOffsetTop = sidebar.offsetTop;
  const offset = stickySidebarOffsetTop - sidebar.parentElement?.offsetTop;

  sidebar.style.position = "relative";
  sidebar.style.top = `${offset}px`;
}, [sidebarRef]);

Now the most tricky one but its name is very self-explainable. Basically, we apply position: relative combined with a proper top to make the sidebar scroll. To perform this math to get the top value, we grab the parent element (aside) offset top and subtract it from the sidebar offset top. It's important you follow our basic HTML structure because you must have a grid item as a parent element around the <Filters /> so it takes the same height as the grid does.

The next step is to start implementing the calculateAndApplyPosition function. Again, we'll need to do some math here and use the results as conditionals to apply the methods we defined above. We also need to create a new ref to store the scroll position:

const lastScrollY = useRef(0);

const calculateAndApplyPosition = useCallback(() => {
  const sidebar = sidebarRef.current;
  const { pageYOffset, innerHeight } = window;

  const isSidebarTallerThanViewport = sidebar.offsetHeight >= innerHeight;

  const hasReachedSidebarEnd =
    sidebar.offsetTop + sidebar.offsetHeight <=
    Math.round(pageYOffset + innerHeight);

  const isScrollingDown = lastScrollY.current < pageYOffset;
  const isScrollingUp = !isScrollingDown;
  lastScrollY.current = pageYOffset;

  const hasReachedSidebarStartByScrollingUp = pageYOffset <= sidebar.offsetTop;
}, [sidebarRef]);

Alright, now we just need to trigger our functions based on the conditions:

const calculateAndApplyPosition = useCallback(() => {
  ...

  if (!isSidebarTallerThanViewport) {
    attachSidebarToTheTop();

    return;
  }

  if (
    isSidebarTallerThanViewport &&
    hasReachedSidebarEnd &&
    isScrollingDown
  ) {
    attachSidebarToTheBottom();

    return;
  }

  if (
    isSidebarTallerThanViewport &&
    hasReachedSidebarStartByScrollingUp &&
    isScrollingUp
  ) {
    attachSidebarToTheTop();

    return;
  }

  if (isSidebarTallerThanViewport) {
    makeSidebarScroll();

    return;
  }
}, [sidebarRef]);

Check out the result 🤩

However, we have a bug when we reach the end of the page and the sidebar gets bigger somehow, like expanding one of its filters:

In order to fix that we will need another method specific for this purpose:

const lastSidebarHeight = useRef(0);

const fixSidebarOverflow = () => {
  const sidebar = sidebarRef.current;

  if (!sidebar || !sidebar.parentElement) return;

  const parentOffset = sidebar.parentElement.getBoundingClientRect().bottom;
  const sidebarOffset = sidebar.getBoundingClientRect().bottom;

  const overflowDetected = sidebarOffset > parentOffset;

  if (overflowDetected) {
    const diff = sidebar.offsetHeight - lastSidebarHeight.current;
    makeSidebarScroll(diff);
  }

  lastSidebarHeight.current = sidebar.offsetHeight;
};

Again we have to use some math here. First, we get how far the bottom of the sidebar parent is from the top . Then we do the same thing for the sidebar and check if one value is bigger than the other. If so, there's an overflow happening, which means that we can trigger our fix function and update the lastSidebarHeight ref. Notice that we pass a diff argument to the function and it's necessary to recalculate the sidebar top distance properly. Let's refactor the makeSidebarScroll to accept this param:

const makeSidebarScroll = useCallback(
  (diff = 0) => {
    const sidebar = sidebarRef.current;
    const stickySidebarOffsetTop = sidebar.offsetTop;  
    const offset = stickySidebarOffsetTop - diff -sidebar.parentElement?.offsetTop;

    sidebar.style.position = "relative";
    sidebar.style.top = `${offset}px`;
  },
  [sidebarRef]
);

Since it only happens when the sidebar size changes let's use the use-resize-observer to listen to this change and then trigger a handler. Go ahead and install it, then instantiate the hook passing the ref and a handler that calls the method we created previously:

const onTargetResize = () => {
 fixSidebarOverflow();
};

useResizeObserver({
 ref: targetRef,
 onResize: onTargetResize,
 wait: 0,
});

Tests

I don't think it's possible to write tests for this hook with testing-library due to the jsdom limitations. I recommend you use cypress, but I won't be covering it in this article.

Final thoughts

The UX of our services is very important and we should always look attentively trying to improve as much as possible. Especially in e-commerce where the final goal is to convert users into buyers, we could have lost a lot of them just because they didn't find the filters to locate their products faster. I created a repository with the final code on my GitHub, check it out.

Thanks for reading ❤️