DEV Community: Summerbud

My problem about telling others to blindly read source code

Summerbud — Fri, 22 Sep 2023 22:00:00 +0000

“You should read the famous repo’s source code. Just read them, you will grow” I keep hitting this suggestion lately and it’s bothering me.

My problem with this suggestion is without any goal in mind, blindly reading source code won’t help and will even cause harm. You need to have a goal in mind to make the magic work. Like “Hey, I want to learn about how to get the header using remark”. For this question, I found out Astro.js has an Astro.glob function, the returns have a getHeadings function. How about I dig inside this function to see how it works?

Another example. Recently, I needed to implement an Auth-Guard middleware, I immediately remembered that Auth0 provided a similar function. So I go to their repo to see how it can help me to implement my middleware and it is indeed very helpful.

To me, the better version of this suggestion would be “When you encounter services that have good quality and open-source, list it down but read their source code when you need them”.

Playwright tips that will make your life easier - including how to select, act and combat flaky test

Summerbud — Tue, 11 Oct 2022 01:37:30 +0000

Recently I began to help our company implement integration tests and encounter dozens of issues that bother me a lot. Including how to correctly assert elements. How to efficiently wait for an element and most important of all, how to combat the flaky tests.

I found out there has not many resources to discuss this kind of topic and I dip very deep into Playwright issues, discussion, and documentation to find these tips. I think they are all very useful and they makes my life easier.

General tips

Use locator!

The locator will activate the auto-wait feature and the retry ability.
The locator will stay up-to-date even if the DOM has changed. The example below shows that the locator of submit button will track the most up-to-date element. This makes the test smoother and reduces the possibility of a flaky test.

const locator = page.locator('text=Submit');  
await locator.hover();  
await locator.click();

// This uses ElementHandler
const handle = await page.$('text=Submit');  

// handle will always point to the non-hoverable/stale element 
// which may cause issues
await handle.hover();  
await handle.click();

Playwright auto-wait feature

In short, the auto-wait feature means "Hey, when you are going to perform some behavior like CLICK, FILL or HOVER... etc, we will check whether the element is in the state that can carry on these tasks or not, if the conditions met we will perform the behavior, otherwise we will wait until the timeout exceed." ¹

Before I notice this feature I write some tests like this.

test("this is BAD", () => {
    const submit = page.locator("button", {hasText: "Next"});

    // I will check whether button is enabled even it is always 
    // enbled
    expect(await submit.isEnabled()).toBeTruthy()
    await submit.click();

})

This code is bad not only you are wasting your testing time on a non-necessary assertion but also cause some unreliable consequence in some other scenario like fill in the field. You should rely on Playwright's auto-wait nature and let them do the heavy lifting work.

test("this is GOOD", () => {
    const button = page.locator("button", { hasText: "Next" });

    // Click the button and let playwright do the actionability 
    // checking
    await button.click();

})

Don't test for playwright

Sometimes it feels tempting to make sure every step fulfills your needs like the below.

test("this is not that good", () => {
    const emailField = page.locator("input#email");
    await emailField.fill("example@gmail.com");
    await expect(emailField).toHaveValue("example@gmail.com")

})

This looks normal and even responsible at the first glance. But you are testing for Playwright. Due to the nature of auto-waiting Playwright will check whether the field can input or not. If it is not it will be timeout at the filling step, so I think generally speaking this assertion is not necessary.

Always use Promise.all to avoid race condition

Because we are all in an async function when we utilize Playwright, there will have race conditions if we don't control the flow. ²

test("this will have race condition", () => {
    const next = page.locator("button", {hasText: "submit"});
    await next.click()
    await page.waitForResponse("url")
})

The above snippet showcases a scenario if somehow the URL response is before we click the next button (Playwright is a very fast machine), the await page.waitForResponse("url") will be stall. In order to eliminate this kind of race condition, we should utilize Promise.all.

test("this will have race condition", () => {
    const submit = page.locator("button", {hasText: "submit"});

    await Promise.all([
        // We start waiting for response
        page.waitForResponse("url"),

        // We click the button
        submit.click()
    ])
})

In this way, we can safely make sure we wait for the click and the response is finished. None of them will be left behind.

But you should not put multiple actions in one Promise.all(), It will act like multiple people controlling the keyboard at the same time which causes the un-predictable result. ³

test("this is bad", () => {
    const emailField = page.locator("input#email");
    const nameField = page.locator("input#name");

    await Promise.all([
        page.waitForResponse("url"),
        emailField.fill("admin@gmail.com"),
        nameField.fill("Hi")
    ])
})

Use locator.waitFor if you have the visual indicator

There are dozens of waitFor in the Playwright ⁴ and they are very useful in different contexts. In my journey of implementing the integration tests, I found out that locator.waitFor is very reliable if you have a visual indicator. I will take a submit form for example. The form is very easy, fill in the email, click submit button, send the request to the server, respond with userID, and display a welcome message. The test may look like the below.

test("this have possibility to be flaky", () => {
    const emailField = page.locator("input#email");
    const submit = page.locator("button", {hasText: "submit"});
    await emailField.fill("admin@gmail.com")

    await Promise.all([
        page.waitForResponse("user-end-point"),
        submit.click()
    ])
})

Let's say we have a user endpoint on http://localhost:3000/user. When creating the user we will post request to this endpoint. When getting the user information we will get request to this endpoint too. If we still need to constantly get the user's information on this form or if there has global state management which will constantly utilize this endpoint, Playwright may feel confused.

Take the test below as an example. It will be flaky if in the middle of the Promise.all Playwright did get the response but it's not due to the submission.

The safest way of doing so is by relying on a visual indicator and use locator.waitFor⁵ to wait for the indicator to show up on the screen.

test("this is stable", () => {
    const emailField = page.locator("input#email");
    const submit = page.locator("button", {hasText: "submit"});
    const succeedMessage = page.locator("h3", {hasText: "Succeed"});
    await emailField.fill("admin@gmail.com")

    await Promise.all([
        succeedMessage.waitFor("visible"),
        submit.click()
    ])
})

How to combat flaky test

Examine the root cause

The root cause of the flaky test is diverse. It's hard to list them all in this article. But I can share some tips about how to find them.

Use playwright debug mode

There are two different kinds of debug modes in Playwright.⁶

npx playwright test --debug
- This is the killer feature of Playwright, it will display a debug inspector to let you observe what the browser actually did in every step. Use this mode to check whether your locator is correct!
PWDEBUG=1 npx playwright test
- In short, What PWDEBUG mode does is "Hey I will run the browser in headed mode and disable the timeout, so the browser won't close and your test flow won't be interrupted." This will come in handy if you just want to make sure every step of your test is correct. ⁷

Pick one that suits your needs, but remember in --debug mode your click action is much slower than the real testing actions some tests may pass in --debug mode but fail when you run the test without it.

Utilize --repeat-each to observe flaky test

The way to observe a flaky test is by repeatedly operating a test. You can achieve that with npx playwright test --repeat-each=<time_you_want_to_run>. When you find a set of tests is flaky, I recommend using this flag and focusing on one test at a time. ⁸

You could point to the specific test by using a colon and a line number. Take the snippet below as an example you could specifically test with itnpx playwright test form.spec.ts:13

test("this is stable", () => { // This is at line 13 of form.spec.ts
    const emailField = page.locator("input#email");
    const submit = page.locator("button", {hasText: "submit"});
    const succeedMessage = page.locator("h3", {hasText: "Succeed"});
    await emailField.fill("admin@gmail.com")

    await Promise.all([
        succeedMessage.waitFor("visible"),
        submit.click()
    ])
})

Use page.on(”console”) to load useful information

You may have many console.log around your code to make you know better what happened in the frontend. You can use this line of code to log them in the terminal.

page.on("console", (m) => console.log("BROWSER: ", m.text()));

Make sure every steps are correctly waited.

I found out this is usually the number one factor that causes a flaky test. Just like I mentioned above, utilize Promise.all() to correctly wait for the test.

Utilize your trace

Playwright provides --trace flag to let you know what happened during the test. It will record every console.log, network activity, and the action playwright conducted. The recommended way of doing the trace is only to record it when the first test is failing. Generally speaking, this is enough, but when you are facing a flaky test you will record the passed test (by definition flaky test means the first test failed but the second test passed) which is not helpful. ⁹


// Retry when the test failed
import type { PlaywrightTestConfig } from '@playwright/test';

const config: PlaywrightTestConfig = {
  retries: 1,
  use: {
    trace: 'on-first-retry',
  },
};

export default config;

You could set the trace to on and keep the retry to one. In this way, you could observe the flakiness of the test and the trace of the test at the same time.

// Retry every test
import type { PlaywrightTestConfig } from '@playwright/test';

const config: PlaywrightTestConfig = {
  retries: 1,
  use: {
    trace: 'on',
  },
};

export default config;

Test is natively fast, and it is really fast

Disable some crucial action if your request hasn't finished yet. Let's say you have a form that needs to fetch a set of information to correctly construct a dropdown. In a normal situation, that request only takes a blink of time and the user won't even notice the dropdown's data is not complete.

test("this is stable", () => { // This is at line 13 of form.spec.ts
    const emailField = page.locator("input#email");
    const submit = page.locator("button", {hasText: "submit"});
    await emailField.fill("admin@gmail.com");

    // Playwright may timeout in this action due to the request is not finished yet.
    const dropdown = page.locator("select#dropdown");
    await dropdown.selectOption("option-fetched-from-backend")

    const succeedMessage = page.locator("h3", {hasText: "Succeed"});


    await Promise.all([
        succeedMessage.waitFor("visible"),
        submit.click()
    ])
})

In this scenario, your test may be flaky because the Playwright is too fast (You will certainly pass this test in debug mode) It only takes 100ms to finish the actions before selecting the dropdown and it begins to interact with the dropdown. If the request somehow is slower this time this test will fail because there doesn't have any data exist.

The solution is to disable your dropdown when the data is not ready. With the auto-waiting nature of Playwright, it will wait a set amount of time, you can even expand the timeout specific to this action. ¹⁰

Understand your backend

If your integration test involves interaction with the backend, some of the flakiness may come from there.

Check whether your backend can take multiple request at a time

If your backend can only handle a single request at a time you should disable the parallel nature of Playwright. In your configuration file, you could achieve this through this configuration. ¹¹

const config = {
    /* Opt out of parallel tests */
    workers: 1,
}

Check if your operation need to be sequence

If you want your test run in sequences you could leverage the native support of serial tests.

test.describe.serial("This will be a test run in sequence", () => {
    test("test 1", () => {});
    test("test 2", () => {});
}

Or you could separate the test into different test files as a "Test List File", disable the parallel operation of Playwright like the above, and import these files in order. ¹²

Check if your backend have special issue

In our backend, when we create a model we need to fetch the target model from GitHub or HuggingFace which will take an extra amount of time. Not only it take longer to finish, but it will also be affected by the internet condition at that moment. These all add up quickly and make the test exceed its timeout limit.

You could have several ways to solve these problems.

Make some operation timeout much longer. Almost all playwright locators, waitfor, and expect have fine-grained timeout option. You can leverage this option to extend the timeout.

test("this have possibility to be flaky", () => {
    const emailField = page.locator("input#email");
    const submit = page.locator("button", {hasText: "submit"});
    await emailField.fill("admin@gmail.com")

    await Promise.all([
        page.waitForResponse("user-end-point"),
        submit.click()
    ])
})

Mock the data in the backend and don't test for the 3rd party interaction. I think this one is better compared to the first one. You could set up an env variable flag that indicates the whole backend session is for integration test and mock the model that takes extra time to create due to they are connecting with 3rd party source. In this way, your test will be quicker and more reliable.

Sum up

I think there are three rings that rule them all.

Use Promise.all to avoid race conditions.
Disable your field when you haven't finished fetching data.
Use the locator!

Alright, that is enough for today. Playwright is a very good tool. Hope you find the power of it.

The spiral learning method, the beauty and the ugly

Summerbud — Fri, 30 Sep 2022 01:48:07 +0000

There are hundreds of learning methods that each have their strengths and weaknesses. but none of them suit me well enough like this one. I have adapted and developed it through my career as a programmer. I call it the spiral learning method(abbr. Spiral), borrowing from one of the casual chats with my friend. He used this metamorph to describe the way I learn and claim that it is very suitable to build new things.

In short, what the Spiral does is learn when you NEED. Progress accordingly like climbing a spiral stair. Look up your GOAL which needs to be an achievable object.

The origin of this method partially comes from my rebellion toward the learning method which our education adapts, I call it the linear learning method(abbr. Linear). Before we go into the details about the spiral let's talk about the linear way of learning things. I think the linear way of learning things does more harm than good to us.

The critic of linear learning

The Linear is usually how the school teaches us to learn. It has pre-built guidelines for everyone. It becomes quite obvious when you take a look at how we structure our school grades with different progression. With this kind of structure ahead they claim that people can't reliably jump across different subjects and gain insight better than the child who learns things step by step.

I think this is a big blind point in our education. It's true at some point of learning and partially true if you want to enter the route of the academy. But when it comes to how to apply knowledge and make something useful I feel the Linear will fall behind the Spiral in three ways.

Emphasizes knowledge has a pre-set structure

The Linear emphasizes that the knowledge has a pre-set structure and the best way to acquire it is to grab it as a whole. But in reality, to accomplish or build something useful is not about knowing everything but knowing what you need and where to learn them.

The Linear will cause people to think outside of school that there should be a similar structure to guide them. Something like Developer Roadmap¹. But usually, there is no such thing that exists. The knowledge is clustered. We need to cultivate the power to recognize what to learn and where to find them. Sadly our education fell short at this part.

The one fit for all end goal - learn knowledge as a whole

The Linear has a pre-set end goal for everyone, that is learning knowledge as a whole. It's like when reading a novel, you have to finish it chapter by chapter. I think this end goal is not suitable for the rapidly changing environment, especially in programming. Besides that, not everyone will be motivated by the idea of learning knowledge as a whole. People should find a goal that supports their enthusiasm well.

The illusion that knowledge is fixed and doesn't change

The Linear builds up an illusion that knowledge is fixed and it doesn't change at all. But in reality, every piece of knowledge is constantly changing and changing fast.

The Linear makes people lose their curiosity and enthusiasm toward new knowledge. It nurtures generations of people who are not going to learn new things once they enter their career.

In order to tackle this issue it may need a full scale of renewing our education and the learning journey after leaving the school. I think the Spiral is a good start. It is very good at inspiring and enlightening enthusiasm.

In order to let you understand the difference between the Linear and the Spiral. I will showcase how they work in practice.

Work in practice

The linear learning method

According to the Developer Roadmap, a beginner needs to learn "What is the Internet" first, then he can learn the three big giants in modern web development that is HTML, CSS, and Javascript. These are all big topics and may take months to knock on the door and years to master.

After digging a little bit, the beginner can pick up the knowledge of version control and CSS processor then he can begin to learn the framework. The Web development framework usually describes lots of philosophy and thinking along with the documentation especially the React. Newcomers can read through these materials first then dive into the general components of these frameworks.

Once the beginner learns some basic operations of the framework then he can try to think about what he wants to build.

The spiral learning method

First, set up your goal. What drives you to web development? Drop by a fascinating website or feel passionate about gaming and would like to build a mini-game? Choose the goal that can enlighten your enthusiasm the most. In this section I will take building a personal blog for example.

There are a bunch of tutorials about building your blog on Youtube. Pick one that suits your needs. Hopefully, they cover how to set up your development environment too. Follow the tutorial and finish it. In this way you could have a further understanding of what web dev looks like.

Last, demo what you accomplish with your friend and family, ask for their feedback and move on. At this point, you may have a general understanding of HTML, CSS, and some frameworks. You will find out most of your knowledge is wrong but it's totally fine! Just set up a new goal and keep learning.

At this point, I guess you could observe the difference between these two learning routes. The Linear has a pre-set learning route and encourages beginners to learn stuff step by step. The Spiral method challenges the linear way of learning and emphasizes the philosophy "Only learn things that can push the progression of your goal".

The methodology

In short, what the Spiral does is learn when you NEED. Progress accordingly like climbing a spiral stair. Look up your GOAL which needs to be an achievable object.

There have three sub-methodology of the Spiral:

You need to set up a clear and objective goal for your learning. Try to stay at the very edge of your ability and don't exceed too far. Judge your progression and the Effectiveness thought whether you achieve your goal.
Progressively learn stuff that can help you achieve your goal and cultivate the ability to find where to learn them.
When facing a set of knowledge, you don't need to learn them all. Just learn the stuff that pushes the progression of your goal.

Let's dive into the details!

How to pick up your goal?

This question is not about "How to find your goal", that is a very different question compared to this one. I am talking about a situation in which you may have multiple goals or ideas hanging around your mind and it's hard to find which one can bring the most benefit to you. If this is the case you could follow this step to find it out.

Please arrange these ideas into three categories with the difficulty to achieve them. And try to begin with the middle difficulty group.
In the middle difficulty group, pick a random idea and examine it yourself with the following question: Can you imagine a blueprint that can help you achieve this goal? If you can fully imagine a blueprint that means you are familiar with this topic. In this case, we should leave this idea behind because you need to stand at the edge of your ability to maximize the efficiency of learning. You should try to find a goal so that it's quite easy to imagine its route at the start but remain a mystery afterward.
Next! you may have a handful of goals and it's hard to choose which one again. I would suggest you choose a goal that you are most passionate about. In the end, enthusiasm is the most durable tactic to keep you learning.

Once you pick up your goal we can start the journey.

Glasp full picture first, then deep dive

To learn efficiently, you need to have some rough feeling about the direction. In other words, you need to have a map on hand. It's like a Minecraft map with minimum exploration which indicates where you are. Remember the way we learn how to build a blog? We go to Youtube and find some useful tutorials. At this point, you need to find something like it that explains the big picture of your ongoing journey. Some sort of full tutorial will greatly help you. (If your goal is too big you need to cut it into small pieces.)

But sometimes they don't have this kind of tutorials. In this case you may need to find them in some articles, someone else's GitHub repository or from a book. Anything that can help you grasp the full picture will drastically push your progression.

After you grasp the full picture. You could begin to dive deeper. Build much more complicated stuff and learn more.

The caveat of the spiral learning method

I won't say the Spiral suits everyone. I had personally taught some friends and my brother this method and some of them failed. So this part of the article mentions the caveats I learned from my journey.

The general challenges

You will constantly feel overwhelmed because knowledge is overloaded and it is hard to find a good mentor or guideline. Most of the time you need to go through this path alone. But if you could find a companion it would help.
You may find out from time to time there are some errors or pitfalls that keep showing up. That will be the point you need to examine the correctness of your learning. Try to slow down a bit, learn this particular knowledge as a whole and correctly understand it. Try to leverage the Feynman method at this point.
It is constantly struggling and you need to make yourself enjoy it. Focus on the joy when you crack the hard question will help and don't forget to celebrate the small victory. You could buy yourself a drink or have a great dinner with your loved one.

Drive by pure curiosity

The Spiral is all about goal setting and achieving. But there is another great force that will push people to learn, curiosity. During my implementation and carving of this methodology, I find out it's hard to learn something when I can't imagine what is the goal I want to achieve with it.

For example, I am learning Scheme language with the help of The Little Schemer. Great book, no doubt. But I feel demotivated very hard during the learning process. Because I can't imagine what I can achieve with this language.

The Spiral is like a double-edged sword. You need to balance it with another method to have a healthy mindset toward knowledge.

When the goal is too big but you really want to finish it

There is another disadvantage of theSpiral. Once you come to a point where what you learn from a simple goal can't sufficiently fulfill your curiosity. But if you want to leap over the deeper water, you may find out that the scale of the goal may be too big to accomplish.

At this moment I would like to congratulate you first. You have to walk through all the harshness of this learning method and come to the sweet spot.

I would say the end game of learning is to come up with something new, something nobody had thought about. Once you enter this stage you will discover the skyline becomes clearer. You could find somebody to build with you or just cut the goal into small pieces and finish them one by one. Either way, you are approaching something extraordinary.

Extra passion overcomes ordinary

Boiled to the bottom line, passion is the key to learning new things. I would like to write another post about how to hack your passion but I will share my viewpoint here first.

Treat your passion like a campfire, you need to burn something to sustain the fire. In the context of human beings that will be the reward. So remember to reward yourself every time you accomplish something no matter the size of it. But there is some research that indicates inner reward is better than outer reward, so it will be better if you could find joy from inside along the way.
Try to find a concept that people around you tend to not agree with. Dig deeper and find the reason people overlook the concept then try to prove them wrong. I find this very useful to sustain my passion.
Know yourself better with some tools. For example, you could take the VIA Character Strengths Survey to know your strength and what gives you purpose.

The Spiral is like a self-generated/motivated wheel. Once you finish a goal and move on to the next one. You will find yourself having the confidence and the endurance to test the limit and gain more power from it.

This method greatly changes my life and makes me a better learner toward this fast-paced world. I hope you can find some point that you can utilize and practice in your daily life. May the force be with you.

Developer Roadmap ↩

Don’t over emphasize the importance of the react folder structure, focus on the philosophies beyond the folder structure

Summerbud — Tue, 06 Sep 2022 03:51:11 +0000

Managing proper react component folder structure is hard, it needs to be flexible, extendable, and most important of all, it needs to make naming a component as easy as possible.

I adapted several react folder structures across 3 different repo this year, what I discovered is not a steady structure but some philosophies behind the structure.

Structure should be consistent and the principle should be documented.
Component's materials should stay near to the component itself.
It's a good practice to export your component to the outer responsibility layer.
It's a good practice to limit the length of the filename to 3.

I would like to introduce the different structures I used first and then share the context of these philosophies with you.

Pnyx

.
|-- public
|-- src
|   |-- config
|   |-- lib
|   |-- locales
|   |-- pages
|   |-- shared-hooks
|   |-- static
|   |   `-- images
|   |-- types
|   |-- utilities
|   |-- components
|   |   |-- buttons
|   |   |-- layouts
|   |   |-- links
|   |   |-- Footer.tsx
|   |   |-- Header.tsx
|   |   `-- ...etc
|   |-- App.tsx

This is a simple project and most of its pages are static. As a first-time React user, I learned this structure from videos on Youtube. The whole repo was maintained in a flat manner. There was no nested folder with a duplicate name like /components/PageHeader/PageHeader.tsx and I didn’t export any component to its outer layer. Because the codebase is relatively small I didn’t observe much problems during writing this project.

instill.tech@0.1.6

.
|-- .github
|-- .husky
|-- .storybook
|-- .__mocks__
|-- .__tests__
|   |-- ui
|   |   |-- Headline.test.tsx
|-- contexts
|-- .hooks
|-- .lib
|-- markdown
|-- pages
|-- public
|-- stories
|   |-- ui
|   |   |-- Headline.stories.tsx
|-- styles
|-- components
|   |-- forms
|   |-- layouts
|   |-- ui
|   |   |-- blocks
|   |   |-- buttons
|   |   |-- groups
|   |   |-- links
|   |   |-- Headline.tsx
|   |   |-- SubHeadline.tsx
|   |   `-- ...etc

After further exploring and digging deeper, this project has several changes compared to Pnyx.

It adapted storybook and stored stories in a different folder,
It had the test suites and stored the test files in a different folder,
Component folder had categories like form and layouts, it grouped the same components together.

This structure was experimental and I had not formed a steady opinion of React folder structure at that moment. But back then the idea of separating component material(like stories and tests) into different folders caused a lot of trouble. When I refactored some minor changes beneath the component folder, I needed to adjust the /tests and /stories folder too. These folders became inconsistent over time.

Besides, those components were not exported to the outer level, so it was hard to digest them. Every import of a component will occupy a line of code and it looks messy.

import ClientBlock from "@/components/ui/blocks/ClientBlock"
import Headline from "@/components/ui/Headline"

Image some Next.js page component that have 50 to 75 lines of this kind of import and it is very chaotic.

instill-ai/instill.tech@0.13.0

.
|-- .github
|-- .husky
|-- .storybook
|-- docs
|-- public
|-- release-please
|-- src
|   |-- contexts
|   |-- hooks
|   |-- styles
|   |-- types
|   |-- lib
|   |-- pages
|   |-- components
|   |   |-- about
|   |   |   |-- OurMembers
|   |   |   |   |-- OurMembers.tsx
|   |   |   |   |-- MemberIntro.tsx
|   |   |   |   `-- index.ts
|   |   |   `-- index.ts
|   |   |-- career
|   |   |-- docs
|   |   |-- landing
|   |   |-- policy
|   |   |-- ui
|   |   |   |-- Nav
|   |   |   |   |-- Nav.tsx
|   |   |   |   |-- AboutPageLink.tsx
|   |   |   |   `-- index.ts
|   |   |   |-- PageHead.tsx
|   |   |   `-- index.ts

This is the current version of my favored folder structure.

Component's materials are near the component itself.
General purpose components will stay in the components/ui folder.
Non-general-purpose components will stay in the parent's component folder.
- For example, MemberIntro.tsx is a disposable component that will only be used by its parent OurMember.tsx so it should stay in the components/about/OurMember folder not in the components/ui folder. This act can make naming components much easier compared to treating every component as general purpose components and putting them into the components/ui folder.
All components are exported to its outermost responsibility folder. For example, Nav will be exported to the /components/ui folder and the OurMember component will be exported to the /components/about. In this way, the import code will look cleaner.
Under the /components folder, the components are categorized by their responsibility not their functionality. For example, there won’t be a category like /block and /section but /onboarding, /career, and /about. In this way, we could limit general-purpose components in just one place and other components are treated like disposable components under their responsibility folder.

import { Nav, PageHead } from "@/components/ui"
import { OurMember } from "@components/about"

instill-ai/design-system

.
|-- .github
|-- .husky
|-- .storybook
|-- public
|-- src
|   |-- hooks
|   |-- styles
|   |-- types
|   |-- utils
|   |-- index.ts
|   |-- ui
|   |   |-- index.ts
|   |   |-- Buttons
|   |   |   |-- ButtonBase
|   |   |   |   |-- ButtonBase.tsx
|   |   |   |   |-- ButtonBase.stories.tsx
|   |   |   |   `-- index.ts
|   |   |   |-- SolidButton
|   |   |   |   |-- SolidButton.tsx
|   |   |   |   |-- SolidButton.stories.tsx
|   |   |   |   `-- index.ts

This project is a special case when it comes to folder structure. It uses the structure I called "Base and Export" two-level component design.

Base-level components: The foundation of every type of component.
- Base level components won't be exported to be used by exterior consumers.
- Base components will define all the props and functions that the component needs.
Exported-level components: Inherits all the props from a base component but may have a predefined style, meant to be used by the consumers.
- Exported-level components will be consumed with the limited props like onChangeInput, borderStyle, fontFamiliy, or other functional props.
- We will limit as few style props an exported component needs as possible, to control the visual style of the component.

This is a unique way to organize our design system. Overall it brings some fresh air into our projects and serves our needs well.

Philosophies

During this journey, what I learned the most is "There is no one fit for all folder structures". If you encounter someone claiming that, it will be better to leave them there and move on. The reason is quite simple: The technology we use is drastically changing. For example, Next.js has a new layout RFC that will change our folder structure a lot, and the adaption of storybook changes the way we thought about components too. I foresee that we will have more frameworks and libraries like these.

Structure should be consistent and the principle should be documented.

This is my first point. The codebase is not a single person's playground but a collective intelligence that the maintainers had to agree on. No matter what you adopt, you should make the structure as clear and consistent as possible. In this way, other maintainers will have a clear path to look into your project and once you need to refactor the whole repo to adopt a new tech it will be easier to migrate a consistent structure.

Component's materials should stay near to the component itself.

This is the mistake I made at instill-ai/instill.tech@0.6.1 that caused lots of trouble and inconsistency when I tried to refactor its structure. Overall I think we should put our component's materials close to the component itself. Not only to reduce the overhead of refactoring but also to help others find your component. In this way we could help the maintainers to name their components easier too, because they don’t need to come up with a component name that can describe itself in various contexts. It only needs to describe itself under the context of its parent.

It's a good practice to export your component to the outer responsibility layer.

When your file has 30 to 50 imports, it will be messy if the imported function or component are not exported to their outer folder. Imagine you have 30 lines of this kind of code.

import ClientBlock from "@/components/ui/blocks/ClientBlock"
import Headline from "@/components/ui/Headline"

Compared to this way.

import { ClientBlock, Headline } from "@/components/ui"

I think the latter is better.

It's a good practice to limit the length of the filename to 3.

This philosophy comes from a friend of mine. Because we are using the PascalCase for the naming convention of our components, it may be hard to read when the length of words exceeds 3.

CallToAction.tsx and CallToActionBlock.tsx

Don’t over-emphasize the importance of a structure

The ideal structure is changing and changing fast, and it highly relates to the technology we adapt to. I think the best method is to maintain these kinds of fundamental philosophies and adapt new tech and structure in a quick and flexible manner.

^{Photo by Viktor Talashuk on Unsplash}

How to solve: TailwindCSS IntelliSense can't pick up your custom utility classname

Summerbud — Fri, 02 Sep 2022 03:06:38 +0000

Recently, I finally solved a big issue with the TailwindCSS that bother me for a long period of time. That is TailwindCSS intelligence won't pick up the custom utility classnames using the @layer directive.

When you are building the app with TailwindCSS, at some point you need to add a custom style. You could add a custom style within tailwind.config.js file. In this way, TailwindCSS intelligence can correctly pick it up for you.

// tailwind.config.js

module.exports = {
  theme: {
    extend: {
      colors: {
        primary_colour_light_blue: "#40A8F5",
      },
    },
  },
};

But if we want a much more complicated style? The next step comes naturally with the help of a custom utility class using the @layer directive at the CSS file we just injected into the app root.¹

@tailwind base;
@tailwind components;
@tailwind utilities;

@layer utilities {
  ".text-header1": {
    @apply font-sans text-[32px] font-medium leading-[42px];
  }
  ".custom-shadow": {
    boxshadow: "0px 0px 0px 3px rgba(64, 168, 245, 0.2)";
  }
}

And here comes the problem, although this can work, TailwindCSS will not purge this class name and correctly digest it as CSS, the TailwindCSS intelligence can't pick up this class. So when you hover the class itself text-header1 on your IDE, there won't be any style information popping up(issue) ² The reason is quite simple, TailwindCSS intelligence doesn't read through your file and find the injection file, it only read the tailwind.config.js at the root of the project.

So how can I add a custom style that can be picked up by TailwindCSS intelligence? Here comes Tailwind plugin for rescue.

Tailwind CSS plugin

You can use the TailwindCSS plugin to add any custom style you want and intelligence will pick it up for you. The way to add a custom utility class is quite simple.

module.exports = {
  plugins: [
    ({ addUtilities, addComponents }) => {
      addUtilities({
        ".custom-shadow": {
          boxShadow: "0px 0px 0px 3px rgba(64, 168, 245, 0.2)",
        },
      });
    },
  ],
};

You could also add the @apply directive in plugin³

module.exports = {
  plugins: [
    ({ addUtilities, addComponents }) => {
      addUtilities({
        ".test-color-red": {
          "@apply text-red-900": {},
        },
      });
    },
  ],
};

After setting up your plugin, the intelligence can now correctly pick up the custom style, but this is not an ideal solution but only a comfy workaround. I am looking forward to one day, the TaiwlindCSS have native support for this issue.

Here is some working example: instill-ai/console

^{Photo by Jené Stephaniuk on Unsplash}

Build the theme toggle button with Astrojs and tailwindCSS

Summerbud — Thu, 01 Sep 2022 05:12:56 +0000

What Astro had brought to us is a world without Javascript as default. They use this philosophy to reduce bundle size and speed up the first paint of the browser, but it also brings lots of new challenges to the developer, such as toggling the dark theme button on the blog.

With Next.js

I wrote a ThemeContext to toggle dark mode in my blog built on top of Next.js. The ThemeContext will add class="dark" to the root HTML upon mounting the app. In this way, TailwindCSS can recognize what is the current theme.¹

// The code I am using in my blog built with Next.js
import { useEffect, useState, createContext } from "react";

const defaultState = {
  theme: "light",
  toggleDark: () => {},
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q0bxxcx8cz7eea9bmoem.jpg)
};

export const ThemeContext = createContext(defaultState);

export const ThemeProvider = ({ initialTheme, children }) => {
  const [theme, setTheme] = useState("light");

  const rawSetTheme = (rawTheme) => {
    const root = window.document.documentElement;
    const isDark = rawTheme === "dark";
    root.classList.remove(isDark ? "light" : "dark");
    root.classList.add(rawTheme);
  };

  if (initialTheme) {
    rawSetTheme(initialTheme);
  }

  React.useEffect(() => {
    rawSetTheme(theme);
  }, [theme]);

  return (
    <ThemeContext.Provider value={[ theme, setTheme ]}>
      {children}
    </ThemeContext.Provider>
  );
};

With Astro.js

But If we want to remain the usage of Context API, we may need to write something like the below.

// mainPage.astro
---
import ContextWrapperedComponent from "./ContextWrapperedComponent"
---

<ContextWrapperedComponent client:load />


// ContextWrapperedComponent

export const ContextWrapperedComponent = () => {
  // logic for context and components

  return (
    <div>
       // bunch of components that rely on context
    </div>
  )
}

Based on the idea of Astro.js - Partial Hydration², we need to label the whole ContextWrapperedComponent as client:load or client:only to make it hydrated and inject it to HTML as Javascript. But in this way, there are lots of not necessary scripts that got injected and increase the overall bundle size which is not what we want.

So I decided to switch my storage of dark mode state from Context to localStorage.

Upon the first-time mounting, I will access the key's(blog-theme) value under localStorage and determine the page's theme according to this value.
Once the user toggles the theme I will reset the theme of the page and update the value under localStorage.

At the first glance, this implementation looks straightforward, but it still has lots of caveats.

Caveat 1 - Blinking pages

The rendering process of Astro is first sending the HTML and CSS to the client, mounting the hydrated template string, and listening to the event to inject the script. This behavior may lead to some unwanted results.

// This code will update the page's theme upon the finsish 
// of the first time painting, but that is not what we want.

useEffect(() => {
  let theme: "light" | "dark";

  if (typeof localStorage !== "undefined" && localStorage.getItem("theme")) {
    theme = localStorage.getItem("theme") as "light" | "dark";
  } else if (window.matchMedia("(prefers-color-scheme: dark)").matches) {
    theme = "dark";
  } else {
    theme = "light";
  }

  if (theme === "light") {
    setTheme("light");
  } else {
    setTheme("dark");
  }
}, []);

For example, if your default theme is the dark mode but what is stored in localStorage is the light mode. The HTML will first display dark mode, once the client-side is mounted, the event triggers hydrated script injection, your page will be updated as the light mode and this process will keep operating every time the user change pages. As the result, your pages will keep blinking.

To solve the blinking issue, we have to update the theme before the first painting of the browser.

Thankfully, Astro.js have already looked after our back, in their API you could make a script sent to the client with HTML and CSS as-is with the special attribute is:inline³. Astro won't hydrate this script and practice any kind of optimization. To seriously put, this is not welcome under the philosophy of Astro but it is necessary for our application. Besides that, Astro.js's official document has also adopted this approach.⁴

// With this code, we can update the page's theme before the first-time painting.
// You should put the code into <script is:inline>

const html = document.querySelector("html");
const theme = (() => {
  if (
    typeof localStorage !== "undefined" &&
    localStorage.getItem("theme")
  ) {
    return localStorage.getItem("theme");
  }
  if (window.matchMedia("(prefers-color-scheme: dark)").matches) {
    return "dark";
  }
  return "light";
})();

if (theme === "light") {
  html.classList.remove("dark");
} else {
  html.classList.add("dark");
}

Caveat 2 - Transition of the toggle them button

Originally, my toggle dark mode button is under dark mode it will display the sun icon and display the moon icon under the light mode. The issue of this design is close to caveat 1, it will have the blinking issue. At my first thought, I think I can update the icon just like I did with the above solution, but later on, I discovered this didn't work and the reason is quite straightforward.

Because we need the toggle button to be interactive, it needs to prefix with client:load to indicate Astro.js hydrate this script. But upon the first painting, the hydrated script is not injected yet, so the is:inline script can't find the target button to update the icon. The button will only show up once the first painting is finished.

To solve this problem, I need to alter the design of the toggle button. It now displays the sun and the moon icon at the same time but has a glimmer transition to indicate which one is the current theme.

Conclusion

Coding with Astro.js is a fresh journey, it has lots of potential to be discovered, I will keep updating what I found during the exploration.

Why is the documentation of tech products so hard to use? (In the user’s point of view)

Summerbud — Wed, 31 Aug 2022 02:00:34 +0000

Nowadays, our documentation of developer tools is hard to use and we tend to find the solution on other sources such as Youtube, GitHub issues, or blog posts. Its content may easily fall behind or the key point has not been mentioned at all. I consider this issue an emergent problem that we should toggle as soon as possible, and I think two major problems needed to be solved first. (In the user’s point of view)

Documentation is too hard to contribute to
- Documentation easily becomes out of date. People tend to write a blog post rather than contribute to a document because it requires a lot of extra effort without the corresponding incentives.
The context is clustered
- Documentation itself hardly connects to other resources. The context that makes good documentation is clustered.

In this article, I would like to introduce two concepts around documentation that I think are problematic and then further expand it to 3 thinking points toward solving these two issues.

First of all, please recall some documentation you have visited recently and join me with the explanation below. I would like to take Next.js for example which is currently my most visited documentation. Let's start with the "Getting Started" page.

https://nextjs.org/docs/getting-started

Nextjs's documentation has a structure that is popular through various products, let me call this structure "Isolated listing structure". The reason I call them isolated is that they are mainly maintained by the people behind Next.js(although they are open-sourced).

Furthermore, I would like to conclude the mindset behind this documentation as "Owner's bullet-point".

Owner's bullet-point

The reason I call it "Owner's bullet point" is the process of generating documentation is usually accomplished by the product owner. At the start of a product, no one has an understanding better than the owner. The owners are the best person to maintain proper documentation for their consumers and maintainers.

But after a while, the product receives a lot of love and begins to build up the community. It becomes harder and harder to follow up on corner cases and bugs. The owners have to catch up with new commits and solve issues daily, on the other hand, they have to explain new designs, and caveats and provide information for newcomers to overcome these problems. The loading is rising high drastically.

Not every product can catch up, the content begins to fall behind, and some solutions may exist in others' blog posts, StackOverflow's answers, GitHub issues, or discussions. A user needs to connect these solutions with search engines, sometimes the solution on documentation is even wrong.

Isolated listing structure

The isolated listing structure is very common, in e-commerce, in general-purpose websites, on your phone, and in the documentation. It's everywhere.

The structure is mostly arbitrary and opinionated coming from the mindset of "Owner's bullet-point", due to the owner having to first come up with the tree of the structure, in an order that they think is most suitable for their client.

The isolated listing structure is a double-edged sword: To be clear, I am not fully against this structure, it is really helpful in a general context. For example, it's good for initial exploration and if you're familiar with the document, it's quite easy for you to find the information you want.

But on the other hand, it's a fixed structure, it is hard for the structure to evolve and every time the maintainer wants to add something else, it's hard to find an appropriate place if the owner didn't think it well from the beginning. Besides that, users have no other choice but to explore your documentation. They have only one route and it's not enough.

^{Martin Grandjean, CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0, via Wikimedia Commons}

Imagine a dynamic structure, like a force-directed graph (Something like Obsidian can accomplish or a tree diagram, you could find many examples right in their publish section¹). I am not saying force-directed graphs or tree diagrams are better than isolated listing structures. What I want is to encourage everyone not to think in a "listing way" but in a more "dynamic way", so that people can choose whatever suits their needs best. They can use tree diagrams to explore the structure and use the force-directed graph to acknowledge the connection between topics. Or we could come up with a new structure from the ground up which can solve some of these problems.

In combination, the mindset and the structure lead to the problem I think is problematic, listed below.

Documentation is too hard to contribute to

In combination with the arbitrary mindset of "Owner bullet's point" and the "Isolated listing structure", what we have is that current documentation can only be properly maintained by the owner, there is no other easy way to contribute to the documentation. The problem is twofold.

First, based on the "Owner's bullet point" mindset, the author doesn't want some inexperienced person to mess up your documentation and it's quite hard to match the tone of documentation if you don't spend much time staying aligned with your maintainers.

Second, your user doesn't have any incentive to contribute to the documentation, people love your product but if they don't get any credit within the construction of the document or some ownership toward some page of documentation, there is no incentive.

You may argue that they can post an issue and explain their suggestion but the problem remains. The feeling of contributing is very simultaneous, it is just like the feeling of buying something. Imagine a parallel world, in here, if you want to buy something, you need to write down 200 words to describe why you want to buy this stuff and what is the benefit the community.

There is no need to gate people from contributing documentation with bureaucracy. We need to come up with another structure that can remain the authority of the documentation and benefit the simultaneous contribution at the same time.

The context is clustered

The material of good documentation is not just the documentation itself, but the corresponding discussions, issues, related blog posts, and videos. I will call these materials "Contexts".

Until recently we tended to store these contexts distributedly. A regular open-source project stores their discussion in GitHub discussion or a discourse forum, their use cases in product websites, issues right in the GitHub issues, and a self-hosting documentation website elsewhere. Besides that, there is a lot of community-generated content on YouTube and personal blog posts.

In reality, the context of a product becomes clustered. They won't have inter-connection with each other. There may be a discussion about the solution to the specific bug, but there is no reverse link on the documentation section that has mentioned this solution. Sadly we have to deal with the reality that the bi-directional link is not what the current internet is capable of.

Clustered context is volatile

Imagine a situation where you drop in the product's documentation and quickly find out that the documentation doesn't provide a solution, later the day you find a workable solution on someone else's blog post.

At this moment, with the current documentation, there is no other way to remind other developers that they can solve the same problem with this method described by the blog post, you can't add a reference to the documentation. What you can do is open an issue(if it's open-source) or open a discussion on a forum to remind people about this solution which will soon be flooded by other content.

Every useful solution in any context needs to be resistant to the flood of information we have right now. They have to join this battleground with no anchor point besides search engines.

The awesome list ² is a good idea, they provide a way to let good contents stay, but they have the same issue with the "Isolated listing structure" and the "Owner's bullet-point" mindset.

If the situation remains as is…

The immediate consequence of these issues will be the “Documentation” will deem to decay over time. You could take a look at some tech giant’s documentation including Amazon web services documentation or Google cloud documentation, they are all overwhelming and hard to read.

The worst feeling is you are stucking on a specific problem that you can’t find in the documentation and anywhere else. We will be facing these kinds of situations more if the structure of our documentation can not align with the scope of the product we use.

It may seem overwhelming to come up with a new kind of structure to overcome these issues in the first place. But look closely, we could separate the overall problems into 3 questions to ask ourselves.

How to encourage users to contribute to the documentation and not interfere with the general purpose of the document? Can we have a more interactive experience for documentation? Can a user directly contribute to the documentation without leaving the page?
How to gather context together for a better search experience that does not rely on exterior search solutions and brings up inter-connectivity of every context along the way?
How to experiment with a new structure that may benefit user experience and iterate it over time, or even let users choose it freely?

I want to progressively tackle these issues. If you are interested in or come up with something interesting about this topic, I'd like to hear about it. I would answer any email (summerbud.chiu@gmail.com) about this topic, you could also find me on Twitter (https://twitter.com/EiffelFly).

^{Cover Photo by Wesley Tingey on Unsplash}

Obsidian publish page ↩
sindresorhus/awesome ↩